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Figure  2-1  -  MTADS  GEM-3  (GEMTADS)  array  in  survey  configuration  with  the  MTADS 

tow  vehicle . 2 

Figure  2-2  -  MTADS  EM  trailer  with  approximate  locations  of  GPS  and  IMU  equipment 

indicated.  The  colored  circles  represent  the  GEM-3  sensors  of  the  GEMTADS 
array . 3 

Figure  2-3  -  MTADS  GEM-3  array  mounted  on  the  EM  sensor  trailer.  In  addition  to  the  three 
GEM-3  sensors,  note  the  three  GPS  antennae  and  the  IMU  for  platform  motion 
measurement.  The  sensors  are  false-colored  as  discussed  in  the  text.  The  IMU  is 
the  small  black  box  mounted  under  the  MB2  GPS  antenna . 3 

Figure  2-4  -  Schematic  of  the  interleaved  survey  pattern  for  GEMTADS  surveys.  The  sensors 
are  depicted  as  colored  circles.  The  large  cross-hatched  sections  indicate  the  path 
of  the  tow  vehicle  tires.  The  outer  extents  of  the  swath  of  the  EM  trailer  tires  are 
represented  by  the  narrow  cross-hatching.  The  tan  bars  represent  areas  where  two 


tire  tracks  are  collocated . 5 

Figure  2-5  -  GEM-3  array  control  electronics  and  GPS  receivers  rack-mounted  in  the  MTADS 
tow  vehicle . 5 

Figure  2-6  -  Working  screen  of  the  WinGEM2kArr  program . 6 

Figure  2-7  -  Screenshot  of  MTADS  Pilot  Guidance  application  display . 6 

Figure  2-8  -  Screenshot  of  the  GPS  monitoring  tool . 8 

Figure  2-9  -  Screenshot  of  the  data  processing  working  screen  in  Oasis  montaj . 8 


Figure  2-10  -  Peak  anomaly  amplitude  results  from  the  GEMTADS  and  pit  measurements  of 
the  4.2-in  mortar  (open  diamonds).  The  modeled  system  response  for  the  most 
(red)  and  least  (blue)  favorable  orientations  of  the  mortar  are  shown  as  lines.  The 


responses  for  the  seeded  GPO  items  are  also  shown  as  ‘x’s . 10 

Figure  2-1 1  -  An  example  of  GEMTADS  fit  results  for  the  training  set  data  from  Former 

Camp  Sibert . 12 


Figure  3-1  -  Available  survey  areas  within  the  F.E.  Warren  AFB  Closed  Base  Ranges  MRA. 

The  survey  areas  are  labeled  by  stakeholder  priority.  Figure  provided  by  URS.  18 

Figure  3-2  -  Northwest  -  Southeast  survey  anomaly  map  for  FEW  GPO . 22 
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Figure  3-3  -  Northwest  -  Southeast  survey  anomaly  map  for  FEW  GPO  with  emplaced  items 
indicated.  Note  that  there  are  also  several,  large-amplitude  anomalies  indicated 


that  do  not  correspond  to  emplaced  items . 23 

Figure  3-4  -  Northeast  -  Southwest  survey  anomaly  map  for  FEW  GPO . 23 


Figure  3-5  -  Peak  anomaly  amplitude  results  from  the  MTADS  GEM-3  array  (GEMTADS) 
system  and  measurements  of  the  emplaced  37-mm  projectiles  in  the  FEW  GPO. 
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items  are  shown  as  ‘x’s.  The  RMS  background  level  for  the  GPO  and  the 
USACE  1  lx  rule  are  shown  as  dashed  lines . 26 

Figure  3-6  -  Peak  anomaly  amplitude  results  from  the  MTADS  GEM-3  array  (GEMTADS) 
system  and  measurements  of  the  emplaced  75-mm  projectiles  in  the  FEW  GPO. 
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Figure  3-7  -  GEMTADS  anomaly  map  (ppm,  Qave)  for  Priority  Area  1.  The  original,  planning 
boundary  is  shown  in  pink.  The  As-Surveyed  boundary  is  shown  in  brown.  The 
subset  area  selected  for  detailed  data  analysis  and  for  data  collection  by  other 
demonstrators  is  shown  in  orange . 28 

Figure  3-8  -  Anomalies  (‘x’s)  above  the  detection  threshold  (1.1  ppm,  Qave)  for  Priority  Area  1. 

The  original,  planning  boundary  is  shown  in  pink.  The  As-Surveyed  boundary  is 
shown  in  brown.  The  subset  area  selected  for  detailed  data  analysis  and  for  data 
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Figure  3-10  -  Anomalies  (‘x’s)  above  the  detection  threshold  (1.1  ppm,  Qave)  for  Priority  Area 
2.  The  original,  planning  boundary  is  shown  in  pink.  The  As-Surveyed  boundary 
is  shown  in  brown . 32 

Figure  3-1 1  -  EMI  data  acquired  over  the  GPO  overlain  by  black  circles  identifying  seeded 
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Figure  3-12  -  EMI  data  acquired  over  the  GPO  overlain  by  anomaly  picks  that  are  not 

associated  with  a  seeded  item.  Yellow  circles  identify  picked  targets  that  were 
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Figure  3-13  -  Scatter  plot  of  the  fit  mismatch  versus  coherence  ratio  for  emplaced  UXO  in  the 
GPO  area.  The  red  symbols  identify  anomalies  with  a  fit  error  of  less  than  50% 
and  fitted  depths  of  less  than  0.75  m.  The  blue  symbols  identify  anomalies  that 
exceed  these  thresholds.  See  text  for  discussion . 35 

Figure  3-14  -  EMI  data  acquired  over  the  GPO  overlain  by  circles  identifying  seeded  ordnance 
items.  The  black  circles  identify  targets  with  lit  errors  less  than  50%  and  fitted 
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Figure  3-17  -  Scatter  plot  of  fit  results  for  anomalies  in  PA1  subset  and  GPO  areas.  See  text 

for  description . 37 

Figure  3-18  -  Histogram  showing  the  frequency  of  labeled  UXO  as  a  function  of  the  decision 
metric.  As  shown,  all  labeled  UXO  not  excluded  based  in  SNR  posses  values 
greater  than  -2 . 38 
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ESTCP 

FA 
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(PTNL,)GGK 


Definition 

Air  Force  Base 

U.S.  Air  Force  Center  for  Engineering  and  the  Environment 

Airborne  Multi-sensor  Towed  Array  Detection  System 

Compact  Disk  -  Recordable 

course-over-ground 

Data  Acquisition  (System) 

Data  Analysis  System 
Writable  digital  versatile  disc 
ElectroMagnetic  Induction 

Environmental  Security  Technology  Certification  Program 

False  Alarm 

False  Alarm  Rate 

F.E.  Warren  AFB 

Fix  Quality 

MTADS  GEM-3  sensor  array 
Geophysical  Prove -Out  (area) 

Global  Positioning  System 
Health  and  Safety  Plan 
Hertz 

Institute  for  Defense  Analyses 

Inertial  Measurement  Unit 

Munitions  Constituents 

Munitions  and  Explosives  of  Concern 

Munitions  Management 

Munitions  Response  Area 

Munitions  Response  Site 

Multi-sensor  Towed  Array  Detection  System 

"Missle-X"  or  Peacekeeper  missle  system.  The  missle  is 

officially  designated  as  a  LGM-1 18A 

North  American  Datum  of  1983 

North  American  Vertical  Datum  of  1988 

National  Marine  Electronics  Association 

Naval  Research  Laboratory 

nanoTesla 

Probability  of  Detection 
Point  of  Contact 
Peak-to-Peak 

Time,  Yaw,  Tilt,  Range  for  Moving  Baseline  RTK  NMEA- 
0183  message 

Time,  Position,  Position  Type,  DOP  NMEA-0183  message 
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Abbreviation 

Definition 

QA 

Quality  Assurance 

QAO 

Quality  Assurance  Officer 

QAPP 

Quality  Assurance  Project  Plan 

QC 

Quality  Control 

RI 

Remedial  Investigation 

ROC 

Receiver  Operating  Characteristic 

RTK 

Real  Time  Kinematic 

SHERP 

Safety,  Health,  and  Emergency  Response  Plan 

SNR 

Signal  to  Noise  Ratio 

SSO 

Site  Safety  Officer 

SUXOS 

Senior  UXO  Supervisor 

TBD 

To  Be  Determined 

UTC 

Universal  Coordinated  Time 

UXO 

Unexploded  Ordnance 

VHF 

Very  High  Frequency 

WAA 

Wide  Area  Assessment 

WGS  84 

World  Geodetic  System  1984 

ZIP  (250) 

Iomega  ZIP  disk  (250  MB  version) 
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ABSTRACT 


Nova  Research  conducted  a  geophysical  survey  of  two  9-acre  areas  within  the  boundary  of  the 
F.E.  Warren  AFB  (FEW)  Closed  Base  Range  MRA  as  part  of  the  Environmental  Security 
Technology  Certification  Program  (ESTCP)  Unexploded  Ordnance  (UXO)  Innovative 
Technology  Transfer  Project.  The  survey  was  conducted  using  the  Naval  Research  Faboratory 
(NRF)  GEMTADS  frequency-domain  EMI  sensor  array.  URS  Corporation,  Inc.  has  been 
previously  tasked  by  the  U.S.  Air  Force  Center  for  Engineering  and  the  Environment  (AFCEE) 
to  conduct  a  Remedial  Investigation  (RI)  of  munitions,  explosives  of  concern  (MEC),  and 
munitions  constituents  (MC)  within  the  Munitions  Response  Area  (MRA).  The  two  survey  areas 
were  selected  in  cooperation  with  the  stakeholders  to  foster  a  comparison  between  the  existing 
geophysical  technologies  and  methodologies  being  used  in  the  RI  with  the  GEMTADS  platform 
and  a  data  processing  methodology  recently  demonstrated  as  part  of  the  ESTCP  UXO 
Discrimination  Study  as  a  possible  avenue  of  technology  transfer  from  the  research  community 
into  the  field.  The  WWI  and  WWII  vintage  ranges  within  the  MRA  include  mostly  37-mm  and 
75-mm  projectile  and  small  arms  ranges.  This  demonstration  data  report  serves  to  document  the 
data  collection  results  of  this  demonstration  and  to  provide  an  archive  for  the  collected  data  sets 
and  other  generated  data  products.  In  addition,  results  of  the  data  processing  effort  to  build 
classifiers  based  on  the  available  ground  truth  are  presented  and  the  feasibility  of  their  immediate 
use  for  discrimination  and  the  construction  of  a  prioritized  dig  list  are  discussed. 
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1.  Introduction 


1.1  Background 

In  FY  2007,  the  Environmental  Security  Technology  Certification  Program  (ESTCP)  was 
directed  by  the  United  States  Congress  to  establish  an  Unexploded  Ordnance  (UXO)  Innovative 
Technology  Transfer  Project.  The  project  was  directed  to  conduct  evaluations  of  innovative 
UXO  technologies  at  military  munitions  response  sites  (MRSs)  in  coordination  with  remediation 
project  managers.  In  December  2006,  ESTCP  solicited  nominations  for  sites  interested  in 
hosting  one  of  the  technology  demonstrations.  Over  twenty  nominations  were  received. 

The  ESTCP  Program  Office  selected  three  sites  for  participation  in  this  year’s  efforts: 

Ft.  McClellan  in  Anniston,  AL;  F.E.  Warren  AFB  outside  Cheyenne,  WY;  and  the  Blossom 
Point  Research  Facility,  Welcome,  MD.  At  each  of  the  sites,  the  technologies  to  be 
demonstrated  in  the  ESTCP  UXO  Innovative  Technology  Transfer  Project  were  chosen  to 
complement  and  extend  ongoing  efforts  at  the  site.  The  technologies  scheduled  for 
demonstrations  range  from  cued  discrimination  using  EMI  sensors  to  underwater  ordnance 
detection  using  magnetometer  arrays. 

1.1.1  Specific  Objective  of  Demonstration 

URS  Corporation,  Inc.  has  been  tasked  by  the  U.S.  Air  Force  Center  for  Engineering  and  the 
Environment  (AFCEE)  to  conduct  a  Remedial  Investigation  (RI)  of  munitions,  explosives  of 
concern  (MEC),  and  munitions  constituents  (MC)  on  the  Closed  Base  Ranges  at  F.E.  Warren  Air 
Force  Base  (AFB),  Wyoming  (FEW),  an  active  installation  outside  Cheyenne,  WY.  The  WWI 
and  WII  vintage  ranges  include  mostly  37-mm  and  75-mm  projectiles  and  small  arms  ranges. 
Since  2003,  the  RI  has  consisted  of  transect  and  comprehensive  geophysical  surveys  using 
Geonics  EM6 1  Mkll  sensors  as  well  as  localized  surface  sweeps  and  intrusive  investigation  of 
targets. 

As  part  of  the  ESTCP  UXO  Innovative  Technology  Transfer  Project,  Nova  Research,  Inc. 
conducted  a  geophysical  survey  of  approximately  19  acres  of  the  Closed  Base  Ranges  using  the 
U.S.  Naval  Research  Laboratory  (NRL)  Multi-sensor  Towed  Array  Detection  System  (MTADS) 
GEM-3  (GEMTADS)  sensor  array  as  a  possible  technology  transfer  candidate  to  support  the 
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ongoing  RI.  The  data  were  collected  in  accordance  with  the  overall  Project  goals  and  the 
requests  of  the  stakeholders. 


2.  Technology  Description 

2.1  Technology  Development  and  Application 

The  demonstration  was  conducted  using  the  NRL  MTADS.  The  MTADS  has  been  developed 
with  support  from  ESTCP.  The  MTADS  hardware  consists  of  a  low-magnetic-signature  vehicle 
that  is  used  to  tow  several  different  sensor  arrays  over  large  areas  (10-25  acres  /  day)  to  detect 
buried  UXO.  The  MTADS  tow  vehicle  and  the  GEMTADS  array  are  shown  in  Figure  2-1. 
Positioning  is  provided  using  cm-level  Real  Time  Kinematic  (RTK)  Global  Positioning  System 
(GPS)  receivers.  The  positioning  technology  requires  the  availability  of  one  or  more  known 
first-order  survey  control  points  to  use  as  reference  stations. 


Figure  2-1  -  MTADS  GEM-3  (GEMTADS)  array  in  survey  configuration  with  the  MTADS  tow  vehicle 

2.1.1  GEM-3  (GEMTADS)  Array 

The  GEM-3  sensor  is  a  frequency-domain  EMI  sensor  developed  by  Geophex,  Ltd.  The 
MTADS  GEMTADS  array  consists  of  three,  96-cm  diameter  GEM-3  sensors  in  a  triangular 
configuration  with  two  sensors  across  the  front  of  the  array  and  one  centered  in  the  rear.  The 
nominal  ride  height  of  the  sensors  is  33.5  cm  above  the  ground.  Figure  2-1  show  the  configured 
array  being  pulled  by  the  MTADS  tow  vehicle.  The  roughly  2-m  square  array  is  shown 
schematically  in  Figure  2-2.  Figure  2-3  is  a  close-up  photograph  of  the  array  with  the  sensors 
false-colored  (sensor  #1  shown  in  red,  sensor  #2  in  green,  and  sensor  #3  in  blue).  This  color 
scheme  is  used  throughout  the  remainder  of  the  document  and  in  the  DAQ  software  displays. 
The  array  is  mounted  on  a  rigid  support  which  is  attached  to  the  MTADS  EMI  trailer  using  non- 
metallic  fasteners.  A  plastic  tarp  can  be  added  to  improve  thermal  stability  and  as  protection 
from  precipitation,  when  required. 

Individual  sensors  in  the  array  are  located  using  a  three-receiver  RTK  GPS  system  shown 
schematically  in  Figure  2-2  [1].  The  three-receiver  configuration  extends  the  concept  of  RTK 
operations  from  that  of  a  fixed  base  station  and  a  moving  rover  to  moving  base  stations  and 
moving  rovers.  The  lead  GPS  receiver  (MB1)  receives  corrections  from  the  fixed  base  station  at 
1  Hz.  This  corrected  position  is  reported  at  10-20  Hz  using  a  vendor-specific  National  Marine 
Electronics  Association  (NMEA)  NMEA-0183  message  format  (PTNL,  GGK,  or  GGK).  The 
MB1  receiver  also  operates  as  a  ‘moving  base,’  transmitting  corrections  (by  serial  cable)  to  the 
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next  GPS  receiver  (MB2)  which  uses  the  corrections  to  operate  in  RTK  mode.  A  vector  (AVR1, 
heading  (yaw),  angle  (pitch),  and  range)  between  the  two  antennae  is  reported  at  10  Hz  using  a 
vendor-specific  NMEA-0183  message  format  (PTNL,  AVR,  or  AVR).  MB2  also  provides 
‘moving  base’  corrections  to  the  third  GPS  receiver  (MR)  and  a  second  vector  (AVR2)  is 
reported  at  10  Hz.  All  GPS  measurements  are  recorded  at  full  RTK  precision,  ~2-5  cm.  All 
sensor  readings  are  referenced  to  the  GPS  1-PPS  output  to  fully  take  advantage  of  the  precision 
of  the  GPS  measurements. 


7J 


Figure  2-2  -  MTADS  EM  trailer  with  approximate  locations  of  GPS  and  IMU  equipment 
indicated.  The  colored  circles  represent  the  GEM-3  sensors  of  the  GEMTADS  array. 


Figure  2-3  -  MTADS  GEM-3  array  mounted  on  the  EM  sensor  trailer.  In  addition 
to  the  three  GEM-3  sensors,  note  the  three  GPS  antennae  and  the  IMU  for  platform 
motion  measurement.  The  sensors  are  false-colored  as  discussed  in  the  text.  The 
IMU  is  the  small  black  box  mounted  under  the  MB2  GPS  antenna. 
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An  Inertial  Measurement  Unit  (IMU)  is  also  included  on  the  sensor  array  to  provide 
complementary  platfonn  orientation  information.  The  IMU  is  a  Crossbow  VG300  running  at  30 
Hz.  The  physical  position  of  the  IMU  is  shown  in  Figure  2-3. 

The  standard  GEM-3  sensor  drive  electronics  have  been  modified  to  produce  a  substantially 
higher  transmit  moment  for  this  array.  Each  individual  sensor  can  transmit  a  composite 
waveform  of  one  to  ten  frequencies  in  the  frequency  range  of  30  to  20,010  Hz  with  a  base  period 
of  1/30  sec.  For  this  survey,  a  composite  transmitter  wavefonn  of  nine  frequencies  log-spaced 
from  90  to  20,010  Hz  was  used.  All  demonstrations  to  date  of  the  GEMTADS  array  have  used 
this  composite  waveform.  Two  additional  base  periods  are  required  for  signal  deconvolution  and 
to  output  the  response  from  each  sensor.  The  array  can  therefore  operate  continuously  with  one 
sensor  actively  transmitting  while  the  other  two  sensors  are  processing  data  at  any  given  time. 
Allowing  for  a  short  coil  settling  time  between  the  transmissions  from  each  sensor,  an  effective 
array  sampling  rate  of  just  over  9  Hz  is  achieved.  Sequential  transmitter  operation  also  alleviates 
the  need  for  the  orthogonal  survey  mode  employed  for  the  EM61  Mkll  array.  Coupled  with  our 
standard  survey  speed  of  3  mph,  the  result  is  a  down-track  sampling  spacing  of  ~15  cm.  The 
cross-track  spacing  is  50  cm.  An  interleaved  survey  pattern  is  used  to  decrease  the  cross-track 
spacing  to  25  cm  as  depicted  in  Figure  2-4.  In  some  cases,  such  as  at  survey  area  corners  with 
limited  availability  of  area  outside  the  site  to  turn  the  system  around  (e.g.  a  fence  line),  the 
orthogonal  survey  pattern  used  for  the  MTADS  EM61  Mkll  has  proven  more  efficient  and  is 
functionally  equivalent. 

The  GEM-3  sensors  are  controlled  by  a  custom  electronics  package  designed  and  built  by 
Geophex,  Ltd.  It  is  mounted  in  an  equipment  rack  in  the  MTADS  tow  vehicle  as  shown  in 
Figure  2-5.  Overall  control  of  data  collection  is  accomplished  with  a  custom  version  of  the 
standard  GEM-3  sensor  control  software,  WinGem2KArr,  running  under  Windows  2000  on  our 
data  acquisition  computer.  An  example  of  the  working  screen  of  this  program  is  shown  in  Figure 
2-6.  This  software  package  logs  the  data  from  the  GEM-3  sensors,  the  three  GPS  NMEA 
sentences,  the  time  of  the  GPS  1-PPS  pulse,  the  GPS  UTC  time  stamp,  and  the  IMU  data  in 
separate  files  with  a  common  base  survey  name.  The  IMU  is  software  triggered  at  30  Hz  by  a 
custom  software  application  written  at  NRL.  The  data  are  periodically  transferred  to  the  data 
analyst  for  immediate  QC  checks  and  for  further  processing.  Refer  to  Appendix  B,  Section  B.9 
for  the  details  of  the  file  fonnats. 
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Figure  2-4  -  Schematic  of  the  interleaved  survey  pattern  for  GEMTADS  surveys.  The  sensors  are 
depicted  as  colored  circles.  The  large  cross-hatched  sections  indicate  the  path  of  the  tow  vehicle  tires. 
The  outer  extents  of  the  swath  of  the  EM  trailer  tires  are  represented  by  the  narrow  cross-hatching.  The 
tan  bars  represent  areas  where  two  tire  tracks  are  collocated. 


Figure  2-5  -  GEM-3  array  control  electronics  and  GPS 
receivers  rack-mounted  in  the  MTADS  tow  vehicle 
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Figure  2-6  -  Working  screen  of  the  WinGEM2kArr  program 

2.1.2  Pilot  Guidance  System 

The  GPS  positioning  information  used  for  data  collection  is  shared  with  an  onboard  navigation 
guidance  display  and  provides  real-time  navigational  information  to  the  operator.  The  guidance 
display  was  originally  developed  for  the  airborne  adjunct  of  the  MTADS  system  (AMTADS)  [2] 
and  is  installed  in  the  vehicle  and  available  for  the  operator  to  use.  Figure  2-7  shows  a 
screenshot  of  the  guidance  display  configured  for  vehicular  use. 


Figure  2-7  -  Screenshot  of  MTADS  Pilot  Guidance  application  display 

An  integral  part  of  the  guidance  display  is  the  ability  to  import  a  series  of  planned  survey  lines 
(or  transects)  and  to  guide  the  operator  to  follow  these  transects.  In  the  context  of  this 
demonstration,  the  pilot  guidance  display  can  be  used  to  guide  the  operator  to  the  survey  area 
and  provide  immediate  feedback  on  progress  and  data  coverage.  The  display  provides  a  left- 
right  course  correction  indicator,  an  optional  altitude  indicator  for  aircraft  applications,  and 
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color-coded  flight  swath  overlays  where  the  current  transect  is  displayed  in  red  and  the  other 
transects  are  displayed  in  black  for  operator  reference.  The  survey  course-over-ground  (COG)  is 
plotted  for  the  operator  in  real  time  on  the  display.  The  COG  plot  is  color-coded  based  on  the 
RTK  GPS  system  status.  When  fully  operational,  the  COG  plot  is  color-coded  green.  If  the 
system  status  is  degraded,  the  COG  plot  color  changes  from  green  to  yellow  to  red  (based  on 
severity)  to  warn  the  operator  and  allow  for  on-the-fly  reacquisition  of  the  affected  area.  Figure 
2-7  shows  the  operator  surveying  line  30  of  a  transect  plan. 

2.1.3  Data  Analysis  Methodology 

2. 1.3.1  Data  Collection  QC  Workflow 

Each  data  set  is  collected  using  the  WinGem2KArr  software  package.  The  collected  raw  data  are 
preprocessed  on  site  for  quality  assurance  purposes  using  standard  MTADS  procedures  and 
checks.  The  data  set  is  comprised  of  eight  separate  files,  each  containing  the  data  from  a  single 
system  device.  See  Appendix  B,  Section  B.9  for  further  details  about  file  contents  and  formats. 
Each  device  has  a  unique  data  rate.  During  the  data  import  phase  of  the  QC  process,  software 
written  by  SAIC  computes  the  average  data  rate  for  each  file  as  the  file  is  being  processed.  Any 
discrepancies  are  flagged  for  the  Data  Analyst  to  address.  After  the  data  import  and  QC  phase, 
the  data  are  transferred  to  Oasis  montaj  to  locate  and  map  the  data.  As  part  of  the  import  process 
any  data  corresponding  to  a  sensor  outage,  a  GPS  outage,  or  a  vehicle  stop  /  reverse,  are 
defaulted  or  marked  to  not  be  processed  further.  Defaulted  data  are  not  deleted  and  can  be 
recovered  at  a  later  time  if  so  desired.  An  example  of  the  working  screen  for  examining  the  GPS 
data  is  shown  in  Figure  2-8.  Any  long  wavelength  features  such  as  sensor  drift  and  large  scale 
geology  are  filtered  from  the  data  (demedianed).  In  the  montaj  environment,  the  data  sets  are 
subjected  to  further  QC  analysis  by  the  Data  Analyst  and  then  assembled  into  complete  site  / 
survey  area  databases.  An  example  montaj  working  screen  is  shown  in  Figure  2-9. 

2. 1.3.2  Anomaly  Detection  and  Detection  Threshold  Selection 

Anomalies  are  extracted  from  the  composite  site  databases  in  a  manner  similar  to  that  used  for 
the  ESTCP  UXO  Discrimination  Study  demonstration  at  Former  Camp  Sibert  [7].  Any  anomaly 
with  a  peak  magnitude  greater  than  the  determined  threshold  is  identified  and  placed  on  the 
anomaly  pick  list.  The  process  of  selecting  an  appropriate  threshold  requires  information  about 
the  item(s)  of  interest,  the  response  of  the  sensor  to  the  item(s)  of  interest,  and  the  goals  of  the 
demonstration  especially  in  terms  of  the  depth  of  interest.  The  detection  threshold  is  selected 
based  on  the  predicted  peak  anomaly  magnitude  for  the  item  of  interest.  As  the  items  of  interest 
could  be  positioned  in  a  range  of  orientations  and  at  a  range  of  depths,  response  curves  are 
generated  bounding  the  sensor  response  at  the  most  favorable  orientation  and  at  the  least 
favorable  orientation  of  the  sensor  /  item  of  interest  pair  with  respect  to  the  exciting  field  and  as  a 
function  of  depth. 
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Figure  2-8  -  Screenshot  of  the  GPS  monitoring 
tool 


Figure  2-9  -  Screenshot  of  the  data  processing  working  screen  in  Oasis  montaj 
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An  example  is  given  in  Figure  2-10  for  the  GEMTADS  system  and  a  4.2-in  mortar  (107  mm 
diameter),  the  item  of  interest  from  the  Former  Camp  Sibert  demonstration.  For  the  GEMTADS 
array,  the  quadrature-based  metric  Qave  has  been  found  to  be  effective  for  anomaly  detection. 


Qave 


^jiQnOHz  +  QsiOHz  +  QmOHz  +  Ql6\0Hz  +  QsAIOHz) 


The  upper  curve  represents  the  sensor  response  (in  ppm,  Qave)  for  the  most  favorable  orientation 
of  the  projectile  with  respect  to  the  exciting  field  (the  GEM-3  transmitter)  as  a  function  of  depth 
below  the  surface.  The  sensors  travel  an  additional  33.5  cm  above  the  surface.  The  lower  curve 
represents  the  sensor  response  for  the  least  favorable  orientation.  The  results  of  pit 
measurements  made  under  field  conditions  on  site  are  shown  as  black  circles.  A  representative 
noise  level  from  a  nearby  GPO  is  also  shown.  For  this  example,  the  demonstration  design  set  the 
initial  depth  of  interest  to  be  1  lx  the  diameter  of  the  item  of  interest,  or  1.17  m  for  the  4.2-in 
mortar.  At  this  depth,  the  initial  threshold  was  set  to  be  one-half  the  least-favorable  predicted 
response,  or  1 .3  ppm,  Qave  in  this  example  where  the  least  favorable  response  is  predicted  to  be 
2.6  ppm,  Qave.  This  safety  factor  of  two  was  requested  by  the  ESTCP  UXO  Discrimination 
Study  stakeholders. 

As  part  of  the  ongoing  RI  at  the  Closed  Base  Ranges  at  FEW,  there  is  an  available  geophysical 
prove-out  area  (GPO)  on  site  containing  many  of  the  munitions  types  that  have  been  found  in  the 
MRA  included  37-mm  and  75-mm  projectiles.  To  date,  the  37-mm  and  75-mm  projectiles  have 
proven  to  be  the  bulk  of  the  munitions  recovered  during  the  RI  and  will  be  the  primary  focus  of 
this  demonstration. 


The  munitions  were  buried  at  a  variety  of  depths  and  orientations  to  determine  the  site-specific 
detection  thresholds  and  detection  depths  for  the  various  items.  Data  were  collected  with  the 
GEMTADS  array  over  this  GPO  to  establish  the  appropriate  anomaly  detection  threshold  for  this 
site  and  items  of  interest.  The  data  processing  and  the  results  are  discussed  in  Section  3.2.6. 
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Figure  2-10  -  Peak  anomaly  amplitude  results  from  the  GEMTADS  and  pit 
measurements  of  the  4.2-in  mortar  (open  diamonds).  The  modeled  system  response  for 
the  most  (red)  and  least  (blue)  favorable  orientations  of  the  mortar  are  shown  as  lines. 

The  responses  for  the  seeded  GPO  items  are  also  shown  as  ‘x’s. 

2. 1.3.3  Classifier  and  Discrimination  Data  Processing  Workflow 

The  located  demedianed  GEMTADS  data  (position,  orientation,  and  9  data  pairs  (in-phase  and 
quadrature  response  for  9  transmit  frequencies))  surrounding  the  center  of  each  selected  anomaly 
are  extracted  and  submitted  to  the  solver  engine  developed  by  SAIC,  Inc.  for  the  analysis  of  the 
GEMTADS  data  collected  at  Camp  Sibert  as  part  of  the  ESTCP  UXO  Discrimination  Project. 
Inversion  of  GEMTADS  data  is  done  using  a  standard  dipole  model.  The  solver  algorithm  finds 
6  fitted  parameters:  x,  y,  z,  <f>,  6,  i//,  where  x,  y,  z,  are  the  target  coordinates  (m)  and  (f>,  6,  i//,  are 
the  Euler  angles  (deg)  of  the  model  output.  The  associated  best-fit  betas  (three,  ft),  chi-squared 
(y2),  and  coherence  are  also  found  as  part  of  the  inversion.  The  betas  are  the  principal 
components  of  the  induced  magnetization  tensor.  The  £  ar|d  coherence  are  measures  of  the 
quality  of  the  inversion  result  with  respect  to  the  original  source  data.  Follow-on  fits  are  made 
while  constraining  the  three  Ps  to  match  appropriate  given  values  from  a  library  containing 
values  for  items  including  the  items  of  interest.  The  associated  %  and  coherence  values  are 
found  for  the  fit  to  each  library  item,  and  the  ratios  of  (constrained  coherence)  /  (un-constrained 
coherence)  are  calculated  for  each  fit. 

To  prevent  the  solver  from  becoming  trapped  in  local  minima,  the  unconstrained  inversion 
process  operates  in  two  stages.  The  first  stage  steps  through  fixed  z  values  from  the  ground 
surface  down  to  a  maximum  depth  of  1 .75m.  At  each  z  step,  the  best-fit  target  x,y  position  is 
found  using  the  “matrix”  method  in  which  elements  of  the  response  tensor  are  found  through 
linear  regression.  This  method  permits  fast  run  times  and  experience  has  shown  that  it  is  robust 
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against  local  minima.  No  restrictions  are  placed  on  the  tensor  values:  they  may  be  positive  or 
negative,  even  for  quadrature  data  which  are  normally  only  positive.  Each  channel  is  treated 
independently,  so  each  will  have  a  different  apparent  target  orientation. 

The  ten  best  solutions,  judged  by  %  value,  are  used  as  starting  points  in  the  second  unconstrained 
inversion  stage.  Prior  to  submitting  these  data,  the  Euler  angles  must  be  determined.  Previous 
work  has  shown  that  using  arbitrary  angle  values  for  the  initial  conditions  can  lead  to  failure  of 
the  solver  due  to  local  minima.  Therefore,  a  special  optimization  step  is  perfonned  separately  on 
each  of  the  ten  best  solutions.  In  this  step,  “best-fit”  Euler  angles  are  found  through 
minimization  of  off-diagonal  elements  in  the  response  tensor,  across  all  channels.  The  starting 
points  for  this  optimization  are  the  apparent  Euler  angles  from  each  channel’s  tensor.  The 
second  unconstrained  inversion  stage  searches  all  6  fitted  parameter  values:  x,  y,  z,  (f),  6,  i//, 
through  a  downhill  simplex  minimization.  After  cycling  through  all  ten  of  the  start  points  from 
the  first  inversion  stage,  the  best-ever  solution  is  saved. 

It  has  been  found  that  making  an  adjustment  to  the  in-phase  data  improved  performance,  based 
on  available  ground  truth.  The  adjustment  consists  of  removing  the  local  mean  from  the  In- 
phase  data  at  each  measurement  point:  At  each  point,  the  mean  in-phase  value  across  all 
channels  is  found  and  subtracted  from  all  in-phase  data  at  that  point.  A  parameter  search  is  then 
performed,  and  finally,  best-fit  Ps  are  detennined  through  regression  on  the  full,  unadjusted  data 
set. 

For  each  anomaly  processed  in  the  described  manner,  all  fitted  results  are  reported;  including 
position  (x,y,z),  orientation  (three  angles),  polarizability  spectra,  j2  error,  the  correlation 
coefficient  between  model  and  measured  data,  and  a  signal  strength  metric.  Both  unconstrained 
and  library-constrained  inversions  are  performed,  assuming  that  representative  spectra  are 
observed  for  certain  classes  of  objects  (such  as  the  UXO  targets  of  interest  or  unique  clutter 
classes).  The  size  of  an  equivalent  sphere  is  also  estimated  from  the  ps  via  established 
parametric  models. 

Once  the  inversions  of  all  anomalies  are  complete,  the  resultant  fit  parameters,  or  features,  are 
examined  to  identify  those  which  are  appropriate  for  building  the  classifier  for  the  discrimination 
process  and  generating  the  prioritized  dig  list.  The  available  features  are  examined  to  find  those 
features  which  cluster  the  anomalies  into  discrete  groupings  either  as  a  function  of  a  single 
feature  or  as  a  distance  from  a  central  position  in  a  feature  plane.  At  this  point  in  the  process,  a 
site-specific  set  of  training  data  with  available  ground  truth  is  used  to  identify  those  features  and 
clusters  that  correspond  to  the  items  of  interest  and  those  that  correspond  to  other  items  such  as 
clutter.  Using  the  best  identified  classifier  and  thresholds,  the  anomalies  are  then  ranked  in  a 
prioritized  dig  list.  The  prioritized  dig  list  is  similar  in  nature  to  the  ones  provided  for  the  UXO 
Discrimination  Study  where  the  anomalies  are  ranked  into  five  categories:  (1)  high  confidence 
clutter  (don’t  dig),  (2)  moderate/low  confidence  clutter  (dig),  (3)  can  not  analyze  (dig),  (4) 
moderate/low  confidence  UXO  (dig),  and  (5)  high  confidence  UXO  (dig).  The  prioritized  dig 
list  is  then  provided  in  the  form  of  an  Excel  spreadsheet. 

An  example  from  the  ongoing  analysis  of  the  GEMTADS  data  collected  at  the  Fonner  Camp 
Sibert  is  shown  in  Figure  2-11  [3].  The  4.2-in  mortar  was  the  item  of  interest  at  this  site.  Figure 
2-11  shows  a  plot  of  the  coherence  ratio  (constrained  fit  /  unconstrained  fit)  versus  signal 


12 


Khadr  et  al. 


strength,  Qsum,  were  Qsum  is  defined  as  the  sum  of  all  quadrature  frequency  responses  for  the 
anomalies  in  the  provided  training  data.  The  coherence  ratio  and  QSUm  represent  a  possible  set  of 
features  for  building  a  classifier.  A  decision  line  (similar  to  a  ID  threshold)  is  generated  and  an 
example  is  shown  in  the  Figure  as  a  solid  line.  In  this  example,  items  with  a  response  below  200 
ppm,  Qsum  would  be  classified  as  ‘can  not  invert,  dig’  as  the  coherence  ratio  becomes  a  less 
valuable  classification  tool  at  these  low  signal  levels.  The  remaining  items  are  then  classified 
based  on  the  distance  from  the  decision  line  being  used  as  a  measure  of  the  confidence  of  the 
classification.  Those  items  that  are  below  and  furthest  from  the  line  would  be  declared  ‘high- 
confidence  non-targets  of  interest.”  Those  items  above  the  line  would  be  declared  UXO  with  the 
confidence  of  that  classification  again  scaled  by  the  distance  from  the  decision  line.  In  this  case, 
those  items  with  a  coherence  ratio  of  0.95  or  higher  would  be  declared  ‘high-confidence  target  of 
interest.”  To  determine  the  dig-no  dig  threshold,  a  ROC  curve  would  be  generated  and  the 
distance  from  the  decision  line  that  correctly  classifies  all  UXO  (i.e.  Pd  of  1 .0)  including  a  safety 
margin  would  be  determined.  This  distance  would  then  be  the  threshold  for  the  dig-no  dig 
decision. 


10  100  1000  10000 
Qsum  =  Max(localSum(Q  all  freqs)) 


Figure  2-1 1  -  An  example  of  GEMTADS  fit  results  for  the  training  set  data  from 
Former  Camp  Sibert 

The  data  processing  workflow  outlined  above  requires  that  a  validated  set  of  training  data  be 
available  upon  completion  of  the  individual  anomaly  analysis  portion  of  the  data  processing 
work  flow.  These  training  data  are  used  to  identify  the  relevant  features  and  clustering  of  the 
anomalies.  The  only  training  data  available  are  the  GPO  data.  Examination  of  the  GPO  ground 
truth  provided  by  URS  shows  that  there  are  no  non-munitions  related  items,  or  clutter  emplaced 
in  the  GPO. 

2.2  Previous  Testing  of  the  Technology 

The  Chemistry  Division  of  the  Naval  Research  Laboratory  has  participated  in  several  programs 
funded  by  SERDP  and  ESTCP  whose  goal  has  been  to  enhance  the  discrimination  ability  of 
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MTADS  for  both  the  magnetometer  and  EM-61  array  configurations.  The  process  has  been 
based  on  making  use  of  both  the  location  information  inherent  in  an  item’s  magnetometry 
response  and  the  shape  and  size  infonnation  inherent  in  the  response  to  the  time-domain 
electromagnetic  induction  (EMI)  sensors  that  are  part  of  the  baseline  MTADS  in  either  a 
cooperative  or  joint  inversion.  In  all  these  efforts,  our  classification  ability  has  been  limited  by 
the  information  available  from  the  EMI  sensor.  The  EM6  lisa  time-domain  instrument  with 
either  a  single  gate  to  sample  the  amplitude  of  the  decaying  signal  (Mkl)  or  four  gates  relatively 
early  in  time  (Mkll).  The  first  generation  of  the  MTADS  EM61  Mkll  array  was  demonstrated  in 
2001  [4]  at  the  Badlands  Bombing  Range,  SD  with  little  demonstrable  gain  over  the  single  decay 
of  the  Mkl  array.  A  second  generation  of  the  Mkll  array  with  updated  electronics  was 
constructed  in  2003. 

To  make  further  progress  on  UXO  discrimination,  a  sensor  with  more  available  information  was 
required.  The  Geophex,  Ltd.  GEM-3  sensor  is  a  frequency-domain  sensor  with  up  to  ten 
transmit  frequencies  available  for  simultaneous  measurement  of  the  in-phase  and  quadrature 
response  of  the  target.  In  principle,  there  will  therefore  be  much  more  information  available 
from  a  GEM-3  sensor  for  use  in  discrimination  decisions.  However,  the  commercial  GEM-3 
sensor  is  a  hand-held  instrument  with  relatively  slow  data  rates  and  is  thus  not  very  amenable  to 
rapid,  wide  area  surveys. 

ESTCP  Project  MM-0033,  Enhanced  UXO  Discrimination  Using  Frequency-Domain 
Electromagnetic  Induction,  was  funded  to  overcome  this  limitation  by  integrating  an  array  of 
GEM-3  sensors  with  the  MTADS  platform  [5].  The  project  objective  was  to  demonstrate  the 
optimum  system  built  around  the  Geophex  GEM-3  EMI  sensor  that  delivers  the  most 
discrimination  perfonnance  while  retaining  acceptable  survey  efficiency.  A  three-sensor  array 
system  was  designed  around  a  modified  GEM-3  sensor.  The  system  was  built  and  characterized 
in  2002  and  2003  and  then  demonstrated  at  the  Standardized  UXO  Demonstration  sites  at 
Aberdeen  Proving  Ground  and  Yuma  Proving  Ground  in  2003  and  2004.  At  each  of  the  sites, 
the  Calibration  Lanes,  the  Blind  Test  Grid,  and  as  much  of  the  Open  Field  Area  as  was  possible 
were  surveyed.  For  the  Blind  Test  Grid  and  the  Open  Field,  the  ranked  target  picks  were 
submitted  to  the  Aberdeen  Test  Center  for  scoring.  Appendix  A  summarizes  the  performance  of 
the  GEMTADS  array  at  both  sites  as  reported  in  Reference  5.  Reference  6  compares  the 
detection-only  perfonnance  of  both  the  second-generation  MTADS  EM61  Mkll  and  the 
GEMTADS  arrays  to  other  demonstrators  at  both  of  the  Standardized  UXO  Technology 
Demonstration  Sites.  The  MTADS  EM61  Mkll  and  GEMTADS  anays  were  demonstrated  in 
the  Spring  of  2007  as  part  of  the  ESTCP  UXO  Discrimination  Study  at  the  Former  Camp  Sibert 
[7].  Data  processing  and  the  development  of  performance  results  for  the  various  discrimination 
methodologies  of  the  UXO  Discrimination  Study  are  discussed  in  ESTCP  Pilot  Program 
Classification  Approaches  in  Munitions  Response  Final  Report  [8]. 

2.3  Advantages  and  Limitations  of  the  Technology 

On  large  open  ranges  the  vehicular  MTADS  provides  an  efficient  survey  technology.  Surveys 
with  the  magnetometer  array  often  exceed  production  rates  of  20  acres  per  day.  Production  rates 
for  the  EM  systems  range  from  one-half  to  one-quarter  that  of  the  magnetometer  system  to 
maintain  sufficiently  high  data  densities.  UXO  items  with  gauges  larger  than  20mm  are  typically 
detected  to  their  likely  burial  depths.  The  detection  performance  of  MTADS  magnetometer, 
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EM61  Mkll,  and  GEMTADS  arrays  for  the  range  of  munitions  types  and  sizes  emplaced  at  the 
Standardized  UXO  Demonstration  sites  are  documented  in  References  6  and  7  and  the  references 
within. 

In  the  past,  a  human  operator  has  manually  selected  the  data  corresponding  to  individual 
anomalies.  With  the  experience  gained  as  part  of  the  WAA  Pilot  Project  and  the  UXO 
Discrimination  Study,  robust  automated  anomaly  extraction  methodologies  have  been  developed 
to  significantly  reduce  the  operational  burden  on  the  human  operator.  Using  a  site-specific 
anomaly  extraction  threshold  for  the  item(s)  of  interest  which  takes  into  account  any  stakeholder 
guidance,  an  automated  routine  can  extract  the  location  of  all  anomalies  above  the  threshold  in 
an  unbiased  manner  with  no  operator  involvement. 

The  located,  demedianed  GEMTADS  data  (position,  orientation,  and  9  data  pairs  (In-phase  and 
Quadrature  response  for  9  transmit  frequencies))  surrounding  the  center  of  each  selected 
anomaly  are  then  extracted  and  submitted  to  the  solver  engine  developed  by  SAIC,  Inc.  for  the 
analysis  of  the  GEMTADS  data  collected  at  Camp  Sibert  as  part  of  the  ESTCP  UXO 
Discrimination  Project.  Inversion  of  GEMTADS  data  is  done  using  a  standard  dipole  model. 

The  solver  algorithm  finds  6  fitted  parameters:  x,  y,  z,  (f>,  9,  i//,  where  x,  y,  z,  are  the  target 
coordinates  (m)  and  (f),  6,  i//,  are  the  Euler  angles  (deg)  of  the  model  output.  The  associated  best- 
fit  betas  (three,  /?),  chi-squared  (j2),  and  coherence  are  also  found  as  part  of  the  inversion. 
Therefore,  the  operator  efforts  are  focused  on  the  development  of  the  feature-based  classifiers 
used  to  build  the  prioritized  dig  list  based  on  stakeholder  requirements  and  experience,  rather 
than  on  the  mechanics  of  anomaly  detection  and  data  processing. 

The  presence  of  certain  terrain  features  such  as  deep  ravines  without  good  crossing  points,  thick 
clusters  of  trees,  and  other  non-navigable  features  such  as  steep  hill  faces  can  limit  the  areas  that 
can  be  surveyed.  The  presence  of  long  barbed-wire  fences  without  gates  and  deep  ravines,  steep 
hill  and  plateau  faces  without  good  access  points  can  also  slow  survey  operations  by  reducing 
survey  line  length  and  increasing  travel  time  to  traverse  these  obstacles. 
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3.  Demonstration  Design 


3.1  Performance  Objectives 

Performance  objectives  for  the  demonstration  are  given  in  Table  3-1  to  provide  a  basis  for 
evaluating  the  perfonnance  and  costs  of  the  demonstrated  technology.  These  objectives  are  for 
the  technologies  being  demonstrated  only.  Overall  project  objectives  will  be  given  in  the  overall 
demonstration  plan  generated  by  ESTCP.  The  final  column,  ‘Actual  Performance  Objective 
Met?’  will  be  added  in  future  reports. 

Table  3-1  -  Performance  Objectives/Metrics  and  Confirmation  Methods 


Type  of 

Performance 

Objective 

Performance  Criteria 

Expected  Performance 
(Metric) 

Performance  Confirmation 
Method 

Qualitative 

Reliability  and 
Robustness 

General  Observations 

Operator  feedback  and 
recording  of  system  downtime 
(length  and  cause) 

Quantitative 

Data  Collection 

Survey  Rate 

5  acres  /  day 

Calculated  from  survey  results 

Data  Density 

>20  pts  /  m2 

Calculated  from  survey  results 

Percentage  of 
Assigned  Coverage 
Completed 

100%  as  allowed  by 
topography  /  vegetation 

Calculated  from  survey  results 

On-site  Data 
Throughput 

All  data  QC’ed  in  real  time 
and  data  preparation  for 
analysis  completed  on  site 

Analysis  of  records  kept  /  log 
files  generated  while  in  the  field 
and  recorded  completion  times 

Data  Analysis 

Anomaly  Detection 
Threshold 

GPO  survey  data  establishes  a 
detection  threshold  which  is 
sufficiently  above  the  site 
background  level  to  detect  the 
items  of  interest  to  the  depth  of 
interest 

Analysis  of  GPO  data  and 
generation  of  response  curves 
for  37 -mm  and  75-mm 
projectiles  and  comparison  to 
existing  data  on  depth 
distribution  of  items  of  interest 

Data  Quality  of 
Inversion  Results 

Data  from  anomalies  with  low 
fit  errors  (<50%)  can  be 
successfully  inverted  and 
classified 

Comparison  of  signal  strength 
for  emplaced  GPO  items, 
inversion  results  (fit  error),  and 
classification  (ID)  with  no 
clutter  training  data 

3.2  Testing  and  Evaluation  Plan 

3.2.1  Demonstration  Set-Up  and  Start-Up 

F.E.  Warren  AFB  (FEW)  has  served  a  number  of  functions  since  its  activation  as  a  military  base 
in  the  mid- 1800s.  A  brief  history  of  the  munitions  usage  and  MEC  concerns  for  the  Closed  Base 
Ranges  at  F.E.  Warren  AFB  can  be  found  in  Reference  9,  and  is  summarized  here.  U.S.  Army 
outpost  Fort  D.A.  Russell  was  established  at  the  same  location  in  1867.  In  1930,  the  base  was 
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renamed  for  Francis  E.  Warren,  the  Wyoming  senator  who  played  an  important  role  in  the 
development  of  the  post.  During  World  War  II,  the  size  of  the  installation  more  than  doubled  to 
support  new  missions,  including  the  training  of  Army  personnel.  In  1947,  the  base  was 
transferred  to  the  newly  formed  Air  Force,  under  control  of  the  Air  Training  Command.  The 
Strategic  Air  Command  assumed  jurisdiction  over  the  installation  in  1958.  FEW  was  the  first 
base  selected  for  deployment  of  Atlas  D  missiles,  and  became  the  operations  center  for  the  Atlas 
intercontinental  ballistic  missile  in  1960.  In  1965,  the  Atlas  missiles  were  replaced  by  200 
Minuteman  I  missiles.  Minuteman  III  missiles  replaced  Minuteman  I  missiles  in  1975.  FEW  was 
assigned  to  the  Air  Combat  Command  in  June  1992,  and  then  to  the  Air  Force  Space  Command 
in  July  1993.  At  present,  the  primary  mission  of  FEW  is  to  provide  operational,  maintenance, 
and  security  support  for  150  Minuteman  III  missiles  and  for  the  deactivation  of  the  Peacekeeper 
(MX)  missile  system. 

An  archive  data  collection  effort  was  perfonned  at  FEW  from  23-26  June  2003  to  determine  the 
status  of  past  and  current  range  activities,  as  well  as  to  collect  relevant  environmental  and 
operational  data  on  the  range  areas  identified  [10].  This  data  collection  effort  resulted  in  the 
generation  of  several  historic  range  maps  for  FEW.  A  number  of  small  arms  and  artillery  ranges 
were  identified  from  historical  records  and  are  noted  in  Reference  9.  Two  active  ranges  were 
identified  within  the  Closed  Bases  Ranges  perimeter  and  one  adjacent  to  the  area,  the  Explosives 
Ordnance  Disposal  (EOD),  the  Small  Anns  Firing  Range,  and  the  Trap  Range.  These  areas 
remain  active  and  are  excluded  from  the  RI  activities. 

FEW  and  AFCEE  contracted  URS  Corporation,  Inc.  to  conduct  an  RI  of  the  Closed  Base  Ranges 
beginning  in  2003  to  identify  and  further  delineate  the  MRSs  within  the  munitions  response  area 
(MRA).  The  data  are  being  used  to  evaluate  the  MRSs  for  further  response  subsequent  to  the  RI 
activities. 

As  part  of  the  ESTCP  UXO  Innovative  Technology  Transfer  Project,  Nova  Research,  Inc. 
conducted  a  geophysical  survey  of  approximately  19  acres  within  the  FEW  Closed  Base  Ranges 
MRA  using  the  NRL  GEMTADS  sensor  array.  The  MTADS  vehicular  system  was  mobilized  to 
the  demonstration  site  in  a  U.S.  Navy-owned  53-ft  trailer  by  a  government-contracted 
transportation  company.  The  53-ft  trailer  transported  the  tow  vehicle,  the  sensors  and  sensor 
trailer,  notebook  computers  for  the  analysis  team,  GPS  equipment,  batteries  and  chargers,  office 
equipment,  radios  and  chargers,  tools,  equipment  spares,  and  maintenance  items  to  the  site. 

Some  essential  support  services  were  available  on-site.  Accordingly,  Nova  Research  made 
arrangements  to  acquire  the  remaining  necessary  supplies,  materials,  and  facilities  from 
appropriate  local  vendors. 

A  9’  x  48’  shipping  container,  which  could  be  fully  opened  at  one  end  for  drive-in  access,  was 
mobilized  to  the  site  to  garage  and  for  secure  storage  of  the  MTADS  vehicle  and  sensor 
platform.  The  48’  shipping  container  and  the  53’  trailer  were  placed  in  between  the  two  survey 
areas  on  a  field  road  for  all-around  easy  access.  Power  to  the  trailers  was  provided  by  a  field 
generator  (8  kW)  provided  by  URS.  NRL’s  standby  4  kW  generator  was  available  for  use  but 
was  not  required.  The  power  was  used  to  recharge  the  vehicle  batteries  overnight. 
Communications  among  on-site  personnel  was  provided  by  hand-held  VHF  radios  provided  to 
all  field  personnel.  The  VHF  radio  and  GPS  batteries  were  charged  overnight  in  the  URS  office 
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trailer  and  picked  up  each  morning  on  the  way  out  to  the  site.  Cellular  phone  service  on  site  was 
good  throughout  the  site  and  also  used  for  communications.  Communications  with  the  URS 
teams  operating  on  site  was  maintained  with  a  single  radio  from  one  of  the  FEW  radio  networks 
provided  by  URS.  A  portable  toilet  was  mobilized  to  support  the  field  crew  during  the  field 
effort. 

FEW  is  located  directly  west  of  Interstate  25  outside  Cheyenne,  WY.  The  demonstration  site  is 
located  in  the  middle  to  northern  portions  of  the  MRA.  Three  potential  8.25-acre  (600  ft  x  600 
ft)  survey  areas  were  identified  by  URS  and  prioritized  in  terms  of  value  to  ongoing  efforts  and 
to  the  various  stakeholders.  The  prioritization  is  indicated  in  the  area  labels  (Priority  Area  1  is 
the  first  priority)  and  the  approximate  comer  coordinates  for  the  areas  are  given  in  Table  3-2. 
Figure  3-1  shows  the  MRA  site  boundary  and  the  prioritized  survey  areas.  Priority  Areas  1  and  2 
were  surveyed  during  this  demonstration.  Priority  Area  3,  held  in  reserve  to  cover  any  emergent 
issues  while  the  team  was  in  the  field,  was  not  used.  The  main  criteria  for  selecting  these  areas 
were:  a)  a  target  anomaly  density  of  100-200  anomalies/acre  and  b)  to  generate  useful 
comparison  data  between  methods  currently  being  considered  at  FEW  for  remediation  and  the 
GEMTADS  to  foster  a  potential  technology  transfer  scenario.  Existing  results  for  the  RI  were 
used  to  estimate  the  anomaly  densities.  A  Priority  Area  4  is  shown  in  Figure  3-1  but  was  not  part 
of  this  demonstration. 

Upon  arrival,  the  team  personnel  established  the  base  camp  and  set  up  for  field  operations. 

There  are  eight  GPS  control  points  available  within  the  MRA.  The  details  are  listed  in  Table 
3-3.  The  coordinates  of  the  provided  geodetic  control  points  are  given  in  Table  3-3  (horizontal 
datum:  World  Geodetic  System  (WGS  84);  vertical  datum:  NAVD88).  The  RTK  GPS  base 
station  receiver  and  radio  link  was  established  on  C  ATM  99- 1 ,  one  of  the  available  established 
control  points.  CATM  99-1  was  chosen  to  provide  good  coverage  of  the  GPO  and  both  survey 
areas  and  for  its  easy  access  from  the  access  road  to  the  survey  areas.  The  validity  of  the  control 
point  location  was  verified  by  the  Quality  Assurance  Officer  (QAO)  using  a  man-portable  RTK 
GPS  rover  receiver  to  occupy  the  control  points  CATM  99-3,  Delta,  and  Fox.  Control  point 
CATM  99-1,  occupied  by  the  GPS  base  station,  served  as  the  reference  for  this  test. 

The  measured  locations  of  the  three  tested  control  points  were  found  to  be  within  2-3  cm  of  their 
reported  locations,  verifying  the  internal  consistency  of  the  control  point  network.  A 
discrepancy  discovered  with  the  locations  of  the  GPO  seed  items  is  discussed  in  Section  3.2.5.  A 
USACE  control  point  located  on  a  concrete  pier  was  found  near  Priority  Area  1  and  the  position 
was  recorded  for  future  QC  efforts.  The  monument  marker  ID  field  is  blank  and  an  elevation 
(6,291.45  ft)  is  stamped  on  the  marker.  The  RTK-measured  coordinates  are  given  in  Table  3-3. 

The  EM  trailer  with  the  GEM-3  sensors  was  connected  to  the  tow  vehicle  and  the  system  was 
powered  up  after  being  unloaded  from  the  53’  trailer.  The  connectivity  of  the  sensors  to  the 
DAQ  computer  and  the  establishment  of  nonnal  SNR  performance  were  verified  along  with  the 
operational  state  of  the  vehicle  RTK  system.  Details  of  the  standard  MTADS  calibration 
diagnostic  procedures  are  given  in  Section  3.2. 10.  These  data  were  collected  and  submitted  to 
the  Data  Analyst  for  evaluation.  These  data  were  also  collected  every  field  day  in  accordance 
with  the  procedures  outlined  in  Section  3.2.10. 
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Figure  3-1  -  Available  survey  areas  within  the  F.E.  Warren  AFB  Closed  Base  Ranges  MRA.  The 
survey  areas  are  labeled  by  stakeholder  priority.  Figure  provided  by  URS. 
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Table  3-2  -  Proposed  survey  area  boundaries  for  the  demonstration 


Survey  Area  / 

Corner  ID 

Easting  (m)  |  Northing  (m) 

Survey  Area  / 

Corner  ID 

Easting  (m)  |  Northing  (m) 

UTM  13N,  WGS  84 

UTM  13N,  WGS  84 

Priority  Area  1  NW 

511,213.526 

4,559,893.210 

Priority  Area  3  NW 

510,786.806 

4,560,198.010 

Priority  Area  1  NE 

511,396.406 

4,559,893.210 

Priority  Area  3  NE 

510,969.686 

4,560,198.010 

Priority  Area  1  SE 

511,396.406 

4,559,710.330 

Priority  Area  3  SE 

510,969.686 

4,560,015.130 

Priority  Area  1  SW 

511,213.526 

4,559,710.330 

Priority  Area  3  SW 

510,786.806 

4,560,015.130 

Priority  Area  2  NW 

510,969.686 

4,558,978.810 

Priority  Area  2  NE 

511,152.566 

4,558,978.810 

Priority  Area  2  SE 

511,152.566 

4,558,795.930 

Priority  Area  2  SW 

510,969.686 

4,558,795.930 

Table  3-3  -  Available  Survey  Control  Points  at  F.E.  Warren  AFB 


Point 

Latitude 

Longitude 

HAE  (m) 

Elevation 

(m)* 

Northing  (m) 

Easting  (m) 

WGS  84 

NAVD88 

UTM  Zone  13N,  WGS  84 

CATM  99-1 

41°  10’  51.07547”  N 

104°  53’  08.93029”  W 

1,881.994 

1,897.759 

4,558,840.204 

509,576.923 

CATM  99-3 

41°  10’  30.99830”  N 

104°  52’  43.70502”  W 

1,888.076 

1,903.888 

4,558,221.881 

510,165.473 

ALPHA 

41°  10’  09.28039”  N 

104°  50’  44.66056”  W 

1,872.626 

1,888.624 

4,557,556.557 

512,940.342 

BRAVO 

41°  09’  53.96697”  N 

104°  51’  15.91398”  W 

1,870.574 

1,886.533 

4,557,083.084 

512,212.875 

CHARLIE 

41°  09’  45.01244”  N 

104°  51’  24.59328”  W 

1,862.149 

1,878.099 

4,556,806.619 

512,011.073 

DELTA 

41°  10’  03.70733”  N 

104°  51’  41.62125”  W 

1,872.238 

1,888.154 

4,557,382.469 

511,613.335 

ECHO 

41°  10’  22.40499”  N 

104°  52’  33.65434”  W 

1,876.405 

1,892.236 

4,557,957.219 

510,400.027 

FOX 

41°  09’  43.77514”  N 

104°  52’  43.18168”  W 

1,871.925 

1,887.758 

4,556,765.683 

510,179.697 

Unnamed 
USACE  CP 

41°  11’  24.97426”  N 

104°  52’  00.77172”  W 

1,902.470 

1,918.322 

4,559,887.797 

511,163.253 

*  Provided  elevations  were  used  to  generate  HAE  measurements  using  the  NAVD88/Geoid03  vertical  datum  for  all  points  except 
the  USACE  point.  For  the  USACE  point,  the  measured  HAE  value  (referenced  to  CATM  99-1)  was  used  to  generate  the 
elevation  using  the  NAVD88/Geoid03  vertical  datum. 


The  URS  SUXOS  and  Site  Safety  Officer  (SSO)  provided  a  brief  orientation  to  the  site  for  the 
field  team  the  first  morning  on  site.  The  project  SSO  conduct  a  ‘tail-gate’  safety  meeting  each 
day  that  field  personnel  were  on  site.  The  topic(s)  for  each  day’s  meeting  were  at  the  discretion 
of  the  project  SSO. 

Preventative  maintenance  inspections  were  conducted  at  least  once  a  day  by  all  team  members, 
focusing  particularly  on  the  tow  vehicle  and  the  sensor  trailer.  Any  identified  deficiencies  were 
addressed  according  to  the  severity  of  the  deficiency.  Parts,  tools,  and  materials  for  many 
maintenance  scenarios  were  available  in  the  system  spares  inventory  which  was  located  on  site. 

3.2.2  Period  of  Operation 

A  tentative  schedule  for  the  major  items  in  the  Demonstration  is  given  in  tabular  form  in  Table 
3-4.  All  times  associated  with  survey  activities  are  now  finalized  with  the  completion  of  field 
work.  See  Section  3.2.3  for  a  discussion  of  the  post-field  work  activities. 
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Table  3-4  -  GEMTADS  F.E.  Warren  AFB  Demonstration  Planning  Schedule 


Date 

Planned  Action 

Week  of  August  27th 

Pack  equipment  and  load  the  53  ft.  trailer  at  Blossom  Point,  MD. 

Tue,  September  4th 

53  ft.  trailer  picked  up  at  Blossom  Point,  MD  for  transit. 

Fri,  September  7th 

URS  personnel  received  and  positioned  the  53’  trailer,  first  shipping 
container,  and  portable  toilet. 

Sun,  September  9th 

Personnel  arrived  in  Cheyenne,  WY.  Unpacked  53’  trailer  and  assembled 
/  tested  MTADS  GEMTADS  system. 

Mon,  September  10th 

Surveyed  GPO.  Began  survey  of  Priority  Area  1. 

Tue,  September  11th 

Completed  survey  of  Priority  Area  1.  Began  survey  of  Priority  Area  2. 

Wed,  September  12th 

Completed  survey  of  Priority  Area  2.  Packed  53  ft.  trailer. 

Thu,  September  13th 

Trailer  departed  for  Blossom  Point,  MD.  Personnel  depart  site. 

Mon,  September  17th 

Trailer  arrived  at  Blossom  Point,  MD. 

Week  of  October  22nd 

Submit  Draft  Demonstration  Data  Report  to  ESTCP. 

3.2.3  Scope  of  Demonstration 

Data  collection  was  conducted  at  the  ESTCP  UXO  Innovative  Technology  Transfer  Project 
Demonstration  Site  at  F.E.  Warren  AFB  at  the  request  of  the  ESTCP  Program  Office.  The  site  is 
located  within  the  F.E.  Warren  AFB  Closed  Base  Ranges,  outside  Cheyenne,  WY.  The 
demonstration  site  is  composed  of  two  areas  with  a  total  acreage  of  approximately  19  acres. 

Total  coverage  surveys  were  conducted  using  the  NRL  MTADS  GEM-3  (GEMTADS)  array. 
These  data  were  collected  in  accordance  with  the  overall  Project  goals  and  the  requests  of  the 
stakeholders.  Located,  preprocessed  data  for  each  area  are  delivered  as  a  site  database  archive 
with  this  report.  A  site-specific  anomaly  detection  threshold  was  determined  for  the  two  items  of 
interest  identified  by  the  stakeholders  for  the  site,  the  37-tnm  and  the  75-nnn  projectile.  This 
threshold  was  based  on  data  collected  from  the  onsite  GPO.  For  each  site  database,  all  anomalies 
above  the  selected  threshold  were  identified  and  an  anomaly  list  generated.  A  subset  area  of 
approximately  4.5  acres  was  selected  within  Priority  Area  1  that  contained  approximately  1,400 
anomalies  for  individual  anomaly  analysis  and  for  data  collection  by  other  data  collection 
demonstrators  involved  in  the  project.  The  selection  of  the  subset  area  was  made  in  cooperation 
with  the  Program  Office  after  completion  of  the  field  work.  This  document,  the  Demonstration 
Data  Report,  was  submitted  to  document  the  anomaly  detection  threshold  selection,  the 
individual  anomaly  analysis  results,  and  a  proposed  classifier  and  discrimination  scheme. 

This  Demonstration  Data  Report  includes  the  analysis  of  the  individual  anomalies  within  the 
selected  subset  and  a  discussion  of  the  features  identified  for  building  potential  classifiers  based 
on  thresholds  and  clustering.  A  recommendation  is  presented  for  a  classifier-based 
discrimination  of  the  anomalies  within  the  selected  subset.  The  available  training  data  are 
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limited  to  the  GPO  which  contains  no  clutter  items,  limiting  our  ability  to  definitively 
characterize  the  performance  of  the  proposed  classifier  scheme. 

3.2.4  Operational  Parameters  for  the  Technology 

The  main  operational  parameters  in  this  study  are  the  determination  of  the  appropriate  site  and 
item-of-interest  specific  anomaly  detection  threshold  for  the  GEMTADS  sensor  array,  the 
identification  of  the  features  used  to  generate  the  discrimination  classifier,  and  the  scheme  for 
ordering  the  anomaly  analysis  results  into  a  prioritized  dig  list  based  on  this  classifier.  Data 
collected  on  the  available  GPO  was  used  to  establish  the  anomaly  extraction  threshold.  See 
Section  3.2.6  for  the  discussion  of  how  the  anomaly  detection  threshold  was  selected. 
Additionally,  the  GPO  data  set  was  used  to  fine  tune  the  solver  used  for  the  inversion  process, 
including  the  development  /  refinement  of  library  elements  (Ps)  for  the  37-mm  and  75-mm 
projectiles  to  be  used  for  the  library-constrained  inversions.  See  Section  2.1.3  for  a  discussion  of 
the  discrimination  process.  Based  on  the  inversion  results  and  the  GPO  results,  a  classifier  was 
built  from  the  available  data  features.  The  classifier  and  a  numerical  classification  scheme  was 
then  used  to  generate  prioritized  dig  lists  with  a  ranking  of  (1)  high  confidence  clutter  (don’t 
dig),  (2)  moderate/low  confidence  clutter  (dig),  (3)  can  not  analyze  (dig),  (4)  moderate/low 
confidence  UXO  (dig),  and  (5)  high  confidence  UXO  (dig). 

3.2.5  Geophysical  Prove  Out  (GPO) 

A  geophysical  prove  out  area  (GPO)  was  previously  emplaced  on  site  near  the  two  survey  areas. 
The  emplaced  items  are  munitions  that  are  of  the  types  recovered  on  site  during  previous 
remediation  efforts  including  the  primary  Items  of  Interest,  the  37-mm  and  75-mm  projectiles.  A 
few  recovered  items  were  also  emplaced,  such  as  a  Stokes  mortar.  The  items  were  emplaced  at  a 
range  of  depths  and  orientations  to  explore  the  maximum  detection  depth  of  the  items,  including 
some  at  depths  that  are  likely  beyond  detection  using  current  technologies.  Using  an  EM61  Mkll 
sensor  (Geonics,  Ltd.),  URS  has  established  a  range  of  anomaly  detection  thresholds  for  the 
Mkll  sensor  and  the  munitions  and  MC  found  on  site  under  actual  site  conditions.  Using  these 
anomaly  detection  thresholds,  depths  of  detection  and  Pd  /  FAR  statistics  were  detennined  for  the 
sensor  /  Items  of  Interest  /  site  combination  [9].  We  surveyed  the  GPO  with  the  GEMTADS 
array  to  collect  data  to  aid  in  establishing  the  anomaly  detection  threshold  used  for  this 
GEMTADS  demonstration.  URS  has  provided  the  ground  truth  for  the  GPO  to  us  to  aid  in  our 
analysis.  To  extract  the  most  information  possible  from  the  GPO,  we  surveyed  the  GPO  in  two 
orthogonal  directions  using  the  interleaved  survey  pattern  for  each  direction.  This  was  important 
because  the  GPO  results  are  all  of  the  ground  truth  infonnation  that  is  available  for  developing 
classifiers  for  discrimination.  The  survey  data  and  ground  truth  from  the  GPO  were  used  by  the 
data  processors  to  develop  library  elements  (Ps)  for  the  37-mm  and  75-mm  projectiles  and  to 
fine-tune  the  solver  developed  for  the  ESTCP  UXO  Discrimination  Study  at  Former  Camp  Sibert 
for  the  site  conditions  and  the  Items  of  Interest  prior  to  beginning  the  main  data  processing 
effort. 

A  discrepancy  was  found  between  the  apparent  locations  of  the  emplaced  GPO  items  and  the 
reported  locations  for  the  items.  When  we  surveyed  the  GPO,  we  found  that  the  apparent 
locations  of  the  GPO  items  were  shifted  slightly  from  the  reported  positions.  URS  records 
indicate  that  the  emplaced  items  were  located  using  RTK  GPS  but  with  an  unknown  reference 
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base  station  point.  URS  now  exclusively  uses  satellite-augmented  GPS  (sub-meter)  for  data 
collection.  As  a  result,  we  shifted  the  reported  GPO  item  locations  using  the  apparent  offset 
prior  to  analyzing  our  GPO  data.  The  Berkeley  group  that  was  also  collecting  data  at  FEW  as 
part  of  this  project  was  apprised  of  the  discrepancy.  Until  the  nature  of  the  offset  is  better 
understood,  we  recommend  that  any  further  efforts  use  control  point  CATM  99-1  for  the 
reference  base  station  to  ensure  co-registration  of  data  sets.  The  offsets  are  given  in  Table  3-5. 

Table  3-5  -  Offsets  for  GPO  emplaced  items 


UTM  13b 

f,  WGS  84 

A(Easting) 

-2.219  m 

A(Northing) 

-0.813  m 

The  GPO  data  sets  are  included  on  the  attached  CD.  Figure  3-2  presents  an  anomaly  map  (ppm, 
Qave)  for  the  GPO  when  surveyed  in  a  Northwest  to  SouthEast  direction.  Figure  3-3  is  the  same 
anomaly  map  with  the  emplaced  items  encircled  by  polygons  and  labeled  by  ID.  Note  that  there 
are  also  several,  large-amplitude  anomalies  indicated  that  do  not  correspond  to  emplaced  items. 
Figure  3-4  presents  an  anomaly  map  (ppm,  Qave)  for  the  GPO  when  surveyed  in  a  Northeast  to 
Southwest  direction.  When  comparing  Figure  3-2  and  Figure  3-4,  there  is  clearly  a  difference  in 
the  background  noise  level  and  structure  depending  on  the  survey  direction.  As  will  be  discussed 
in  later  sections,  this  lesson  was  applied  to  the  main  survey.  A  quick  preliminary  survey  was 
conducted  around  the  boundary  of  each  survey  areas  to  determine  the  optimal  survey  direction 
prior  to  data  collection. 
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Figure  3-2  -  Northwest  -  Southeast  survey  anomaly  map  for  FEW  GPO 
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Figure  3-3  -  Northwest  -  Southeast  survey  anomaly  map  for  FEW  GPO 
with  emplaced  items  indicated.  Note  that  there  are  also  several, 
large-amplitude  anomalies  indicated  that  do  not  correspond  to 
emplaced  items. 
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Figure  3-4  -  Northeast  -  Southwest  survey  anomaly  map  for  FEW  GPO 
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3.2.6  Anomaly  Detection  and  Detection  Threshold  Selection 

The  site-specific  anomaly  detection  threshold  for  the  GEMTADS  array  at  FEW  was  determined 
in  the  manner  similar  to  the  one  described  in  Section  2.1 .3.2.  Unlike  our  demonstration  at  the 
Former  Camp  Sibert  [7],  examples  of  each  item  of  interest  were  not  available  for  detailed 
measurements  in  a  controlled  manner  as  a  function  of  depth.  However,  a  GPO  was  available 
which  included  examples  of  the  two  items  of  interest  for  this  demonstration  emplaced  at  a 
variety  of  depths  and  orientations.  The  relevant  portion  of  the  provided  ground  truth  for  the 
GPO  is  given  in  Table  3-6  along  with  the  adjusted  locations,  as  discussed  in  Section  3.2.5. 

Two  sets  of  data  were  collected  using  the  GEMTADS  over  the  GPO  in  orthogonal  directions. 

The  use  of  both  data  sets  doubles  the  number  of  observations  for  each  item  of  interest  to  use  in 
determining  the  system  response.  The  located  demedianed  GEMTADS  data  (position, 
orientation,  and  9  data  pairs  (In-phase  and  Quadrature  response  for  9  transmit  frequencies)) 
surrounding  the  center  of  each  of  the  relevant  emplaced  items  were  extracted  and  submitted  to 
the  GEMTADS  solver  engine.  The  peak  value  of  Qave  for  each  emplaced  item  was  also 
extracted.  The  peak  Qave  values  for  the  emplaced  37-mm  projectile  (14  observations)  and  the  75- 
mm  projectile  (26  observations)  are  shown  in  Figure  3-5  and  Figure  3-6,  respectively,  as  ‘x’s. 

These  values  only  represent  the  system  responses  for  the  observations  made.  To  estimate  the 
true  bounds  of  the  possible  system  response  to  the  items  of  interest,  the  individual  anomaly 
analyses  are  used.  The  resultant  best-fit  betas  (three,  /?)  for  each  item  of  interest  are  used  to 
generate  an  ensemble  average  set  of  Ps  for  each  item  of  interest.  These  ps  were  then  used  to 
forward  model  the  system  response  for  the  item  of  interest  in  terms  of  the  Qave  quantity. 

Response  curves  were  generated  bounding  the  sensor  response  at  the  most  favorable  orientation 
and  at  the  least  favorable  orientation  of  the  sensor  /  item  of  interest  pair  with  respect  to  the 
exciting  field  and  as  a  function  of  depth.  The  response  curves  for  the  37-mm  and  75 -mm 
projectiles  are  shown  in  Figure  3-5  and  Figure  3-6,  respectively.  The  upper  curve  (red) 
represents  the  sensor  response  (in  ppm,  Qave)  for  the  most  favorable  orientation  of  the  projectile 
with  respect  to  the  exciting  field  (the  GEM-3  transmitter)  as  a  function  of  depth  below  the 
surface.  The  sensors  travel  an  additional  33.5  cm  above  the  surface.  The  lower  curve  (blue) 
represents  the  response  for  the  least  favorable  orientation.  The  measured  background  signal 
level  from  the  GPO  is  also  shown  as  a  horizontal  dashed  line.  The  background  signal  level  is 
given  in  Table  3-7.  The  USAGE  1  lx  level  is  represented  as  a  vertical  dashed  line  in  each  Figure 
as  well. 

Based  on  the  modeled  system  response  for  each  item  of  interest,  the  predicted  system  response 
for  the  least  favorable  orientation  at  a  burial  depth  of  1 1  times  the  item’s  diameter  (1  lx)  was 
determined  and  the  results  are  given  in  Table  3-8.  The  least  favorable  response  of  the  37-mm 
projectile  at  a  burial  depth  of  1  lx  was  found  to  be  the  limiting  value  at  1.4  ppm,  Qave.  After 
incorporating  a  safety  factor  of  75%,  the  anomaly  detection  threshold  was  set  at  1.1  ppm,  Qave,  or 
approximately  1.8x  the  site-specific  background  signal  level.  The  appropriateness  of  the 
anomaly  detection  threshold  and  the  site  background  signal  level  were  monitored  throughout  the 
demonstration,  but  no  changes  were  made.  The  individual  anomaly  analysis  results  for  the 
emplaced  items  of  interest  and  the  anomaly  lists  for  the  GPO  generated  using  the  1 . 1  ppm,  Qave 
anomaly  detection  threshold  are  included  on  the  attached  CD. 
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As  a  note,  emplaced  items  FEW-013,  FEW-020,  FEW-029,  FEW-050,  FEW-052,  FEW-053  are 
not  picked  at  the  anomaly  detection  threshold  of  1.1  ppm,  Qave.  Inspection  of  Table  3-6  shows 
that  all  of  these  emplaced  items  except  FEW-050  are  emplaced  at  depths  lower  than  a  detection 
would  be  expected  based  on  the  threshold  selection  criteria.  There  is  no  visible  signature  of  item 
FEW-050  in  either  data  set.  Several  additional  anomalies  are  visible  in  the  anomaly  maps  and 
were  detected  as  well. 

Table  3-6  -  Partial  Schedule  of  Emplaced  Items  in  FEW  GPO 


Target 

ID 

Nomenclature 

Orientation 

Orientation  Details 

Depth 

(ft) 

Northing 
UTM  (m) 

Easting 
UTM  (m) 

Northing 
Corrected 
UTM  (m) 

Easting 
Corrected 
UTM  (m) 

FEW-001 

Projectile,  37-mm 

Vertical 

Nose  Up 

1.00 

4558556.96 

510166.71 

4558556.15 

510164.49 

FEW-002 

Projectile,  37-mm 

Horizontal 

Nose  South 

1.00 

4558556.82 

510142.16 

4558556.01 

510139.94 

FEW-003 

Projectile,  37-mm 

Vertical 

Nose  Up 

1.30 

4558546.60 

510144.45 

4558545.79 

510142.23 

FEW-004 

Projectile,  37-mm 

Horizontal 

Nose  North 

1.30 

4558547.39 

510155.06 

4558546.58 

510152.84 

FEW-005 

Projectile,  37-mm 

Vertical 

Nose  Up 

1.50 

4558562.53 

510149.21 

4558561.72 

510146.99 

FEW-006 

Projectile,  37-mm 

Horizontal 

Nose  North 

1.50 

4558547.59 

510135.95 

4558546.78 

510133.73 

FEW-014 

Projectile,  Shrapnel,  75-mm 

Vertical 

Nose  Down 

1.00 

4558573.99 

510154.09 

4558573.18 

510151.87 

FEW-015 

Projectile,  Shrapnel,  75-mm 

Horizontal 

Nose  East 

1.00 

4558553.01 

510136.99 

4558552.2 

510134.77 

FEW-016 

Projectile,  Shrapnel,  75-mm 

Horizontal 

Nose  East 

2.50 

4558559.37 

510165.20 

4558558.56 

510162.98 

FEW-017 

Projectile,  Shrapnel,  75-mm 

Vertical 

Nose  Down 

1.50 

4558555.29 

510132.36 

4558554.48 

510130.14 

FEW-018 

Projectile,  Shrapnel,  75-mm 

Horizontal 

Nose  South 

1.50 

4558560.97 

510152.09 

4558560.16 

510149.87 

FEW-019 

Projectile,  Shrapnel,  75-mm 

Vertical 

Nose  Down 

3.00 

4558547.43 

510148.07 

4558546.62 

510145.85 

FEW-020 

Projectile,  Shrapnel,  75-mm 

Horizontal 

Nose  Northeast 

3.00 

4558558.20 

510158.75 

4558557.39 

510156.53 

FEW-034 

Projectile,  Shrapnel,  75-mm 

Horizontal 

1.70 

4558559.98 

510141.40 

4558559.17 

510139.18 

FEW-035 

Projectile,  AP,  75-mm 

Vertical 

2.00 

4558559.69 

510146.06 

4558558.88 

510143.84 

FEW-039 

Projectile,  37-mm 

Horizontal 

0.50 

4558538.24 

510143.03 

4558537.43 

510140.81 

FEW-051 

Projectile,  Shrapnel,  75-mm 

Vertical 

Nose  Up 

3.50 

4558582.35 

510164.51 

4558581.54 

510162.29 

FEW-052 

Projectile,  Shrapnel,  75-mm 

Horizontal 

Nose  North 

3.50 

4558573.89 

510166.53 

4558573.08 

510164.31 

FEW-053 

Projectile,  AP,  75-mm 

Vertical 

Nose  Down 

4.00 

4558553.70 

510173.01 

4558552.89 

510170.79 

FEW-054 

Projectile,  AP,  75-mm 

Horizontal 

Nose  East 

4.00 

4558561.23 

510168.96 

4558560.42 

510166.74 

Table  3-7  -  FEW  GPO  RMS  Background  Level 


Array 

RMS  Background  Level 

GEMTADS 

0.6  ppm,  Qave 

Table  3-8  -  System  Response  and  Anomaly  Detection  Thresholds  for  Items  of  Interest 


Item  of  Interest 

Minimum  Response  at  1  lx 

Anomaly  Detection  Threshold 

37-mm  Projectile 

1 .4  ppm,  Qave 

1 . 1  ppm,  Qave 

75-mm  Projectile 

2.7  ppm,  Qave 

2.0  ppm,  Qave 
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Figure  3-5  -  Peak  anomaly  amplitude  results  from  the  MTADS  GEM-3  array  (GEMTADS)  system  and 
measurements  of  the  emplaced  37-mm  projectiles  in  the  FEW  GPO.  The  modeled  system  response  for 
the  most  (red)  and  least  (blue)  favorable  orientations  of  the  mortar  are  shown  as  lines.  The  responses  for 
the  seeded  GPO  items  are  shown  as  ‘x’s.  The  RMS  background  level  for  the  GPO  and  the  EJSACE  1  lx 
rule  are  shown  as  dashed  lines 


Figure  3-6  -  Peak  anomaly  amplitude  results  from  the  MTADS  GEM-3  array  (GEMTADS)  system  and 
measurements  of  the  emplaced  75-mm  projectiles  in  the  FEW  GPO.  The  modeled  system  response  for 
the  most  (red)  and  least  (blue)  favorable  orientations  of  the  mortar  are  shown  as  lines.  The  responses  for 
the  seeded  GPO  items  are  shown  as  ‘x’s.  The  RMS  background  level  for  the  GPO  and  the  USAGE  1  lx 
rule  are  shown  as  dashed  lines. 


GEMTADS  at  F.E.  Warren  AFB 


27 


3.2.7  Priority  Area  1  Results 

The  perimeter  of  the  Priority  Area  1  survey  area,  as  shown  in  Figure  3-1,  is  a  600  ft  x  600  ft  area, 
covering  8.3  acres.  The  actual  area  surveyed  is  somewhat  larger,  covering  9.3  acres  mostly  due 
to  extra  data  collection  along  the  survey  direction  outside  the  original  boundary.  Based  on  the 
lessons  learned  while  surveying  the  GPO,  a  quick  preliminary  survey  was  conducted  around  the 
boundary  of  the  survey  area  to  detennine  the  optimal  survey  direction  with  respect  to 
background  signal  level  prior  to  data  collection.  The  North-South  survey  direction  was  found  to 
be  the  quietest  survey  direction.  The  Qave  anomaly  map  for  Priority  Area  1  is  shown  in  Figure 
3-7.  The  planning  boundary  is  shown  in  pink.  The  As-Surveyed  boundary  is  shown  in  brown. 

A  4.2  acre  subsection  of  Priority  Area  1  was  selected  for  other  demonstrators  to  survey  and  for 
the  discrimination  portion  of  this  demonstration.  The  subset  boundary  is  shown  in  orange.  All 
anomalies  within  Priority  Area  1  with  peak  Qave  values  above  the  1 . 1  ppm  threshold  are  shown  in 
Figure  3-8  as  an  ‘x’  symbol.  3,208  anomalies  exceeded  the  threshold,  and  1,395  of  those 
anomalies  were  within  the  subset  boundary  and  were  further  analyzed.  All  three  boundaries  are 
shown  as  well  using  the  same  color  scheme  as  for  Figure  3-7.  The  located,  demedianed  data 
archive  is  located  on  the  attached  CD.  The  anomaly  list  for  the  entire  area  is  also  located  on  the 
attached  CD  along  with  the  area  boundary  files  in  Geosoft  polygon  file  format  (.ply). 
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Figure  3-7  -  GEMTADS  anomaly  map  (ppm,  Qave)  for  Priority  Area  1.  The  original,  planning  boundary 
is  shown  in  pink.  The  As- Surveyed  boundary  is  shown  in  brown.  The  subset  area  selected  for  detailed 
data  analysis  and  for  data  collection  by  other  demonstrators  is  shown  in  orange. 
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Figure  3-8  -  Anomalies  (‘x’s)  above  the  detection  threshold  (1.1  ppm,  Qave)  for  Priority  Area  1 . 
The  original,  planning  boundary  is  shown  in  pink.  The  As-Surveyed  boundary  is  shown  in  brown. 
The  subset  area  selected  for  detailed  data  analysis  and  for  data  collection  by  other  demonstrators  is 
shown  in  orange. 
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3.2.8  Priority  Area  2  Results 

The  perimeter  of  the  Priority  Area  2  survey  area,  as  shown  in  Figure  3-1,  is  a  600  ft  x  600  ft  area, 
covering  8.3  acres.  The  actual  area  surveyed  is  somewhat  larger,  covering  9.5  acres  mostly  due 
to  extra  data  collection  along  the  survey  direction  outside  the  original  boundary.  A  quick 
preliminary  survey  was  conducted  around  the  boundary  of  the  survey  area  to  determine  the 
optimal  survey  direction  with  respect  to  background  signal  level  prior  to  data  collection.  The 
North-South  survey  direction  was  found  to  be  the  best.  The  Qave  anomaly  map  for  Priority  Area 
2  is  shown  in  Figure  3-9.  The  planning  boundary  is  shown  in  pink.  The  As-Surveyed  boundary 
is  shown  in  brown.  All  anomalies  within  Priority  Area  2  with  peak  Qave  values  above  the  1 . 1 
ppm  threshold  are  shown  in  Figure  3-10  as  an  ‘x’  symbol.  5,223  anomalies  exceeded  the 
threshold.  Both  boundaries  are  shown  as  well  using  the  same  color  scheme  as  for  Figure  3-9. 

The  located,  demedianed  data  archive  is  located  on  the  attached  CD.  The  anomaly  list  for  the 
entire  area  is  also  located  on  the  attached  CD  along  with  the  area  boundary  files  in  Geosoft 
polygon  file  format  (.ply). 
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Figure  3-9  -  GEMTADS  anomaly  map  (ppm,  Qave)  for  Priority  Area  2.  The  original,  planning 
boundary  is  shown  in  pink.  The  As-Surveyed  boundary  is  shown  in  brown. 
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Figure  3-10  -  Anomalies  (6x’s)  above  the  detection  threshold  (1.1  ppm,  Qave)  for  Priority  Area  2. 

The  original,  planning  boundary  is  shown  in  pink.  The  As-Surveyed  boundary  is  shown  in  brown. 

3.2.9  Priority  Area  1  Subset  Data  Processing  Results 

Figure  3-11  presents  the  EMI  data  acquired  over  the  GPO.  Here,  the  quantity  QSUm?  the  sum  of 
the  middle  five  quadrature  channels;  namely,  270,  570,  1230,  2610,  and  5430  Hz,  is  plotted. 
Qsum  differs  from  Qave  by  a  factor  of  5,  Qsum  =  Qave  /  5,  and  was  used  by  the  Data  Analysts  for 
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convenience.  As  detailed  earlier,  49  UXO  have  been  emplaced  in  the  GPO  (3 1  of  the  emplaced 
targets  were  buried  less  than  1  lx  their  diameter,  18  were  buried  deeper  to  characterize  detection 
capabilities).  The  circles  overlain  on  Figure  3-11  identify  the  locations  of  the  emplaced  targets 
of  interest  (TOI).  A  TOI  signature  library  has  been  created  using  28  high  SNR  anomalies  from 
the  individual  anomaly  analysis  results. 
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Figure  3-1 1  -  EMI  data  acquired  over  the  GPO  overlain  by  black  circles  identifying 
seeded  ordnance  items 

The  automatic  target  picker  described  earlier  identified  90  anomalies  that  exceeded  the  detection 
threshold  in  the  GPO  survey  area,  which  includes  all  emplaced  items.  These  additional  anomaly 
picks  are  shown  in  Figure  3-12.  Many  of  these  anomalies  do  not  possess  a  structured  signal  that 
we  nominally  associated  with  buried  UXO,  and  are  therefore  most  likely  bad  picks  that  will 
result  in  false  alarms.  Recall  that  the  anomaly  selection  process  was  based  on  peak  Qave 
amplitude,  and  did  not  evaluate  or  use  expected  signal  structure.  To  reduce  the  number  of 
anomalies  associated  with  transient  signals,  the  dataset  for  each  anomaly  was  sorted  according  to 
distance  from  the  peak  response.  If  seven  of  the  first  N  data  points  (sorted  first  by  distance  from 
peak  response  and  then  by  the  next  highest  value)  possessed  Qave  values  greater  than  the  median 
value  (for  all  data  points  associated  with  the  individual  anomaly)  AND  3  of  the  first  N  data 
points  possessed  values  of  Qave  >  median+rms  (anomaly  dataset  median  Qave  value  plus  the 
anomaly  dataset  Qave  rms),  the  anomaly  was  deemed  to  have  structure  and  therefore  to  be  a  valid 
anomaly  pick.  This  test  was  performed  for  N  values  of  both  10  and  12.  The  yellow  circles  in 
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Figure  3-12  identify  non-seeded  target  picks  that  were  rejected  from  the  list  of  candidate  targets 
based  on  these  rules. 
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Figure  3-12  -  EMI  data  acquired  over  the  GPO  overlain  by  anomaly  picks  that  are 
not  associated  with  a  seeded  item.  Y ellow  circles  identify  picked  targets  that  were 
subsequently  rejected  based  on  statistical  rules  evaluating  peak  signal  strength  and 
local  spatial  variance. 


Figure  3-13  presents  fitted  results  for  the  emplaced  UXO  in  the  GPO.  Here,  we  plot  the  fitted 
mismatch,  defined  as  SQRT(1 -correlation  coefficient^),  versus  the  coherence  ratio.  The 
coherence  ratio  is  defined  as  the  best-fit  library  coherence  divided  by  the  unconstrained  fit 
coherence.  It  ranges  from  0  to  1  where  0  indicates  that  the  anomaly  under  investigation  cannot 
be  modeled  well  using  the  fixed  polarizations  included  in  the  TOI  library.  The  red  symbols  in 
Figure  3-13  identify  anomalies  that  have  fit  errors  of  less  than  50%  and  fitted  depths  of  less  than 
0.75m.  The  blue  symbols  identify  anomalies  that  exceed  these  thresholds.  Anomalies  that 
exceed  these  thresholds  will  be  ranked  Category  3  and  recommended  to  be  dug.  The  solid  blue 
line  limits  to  the  mean  value  of  the  coherence  ratio  for  high  SNR  seeded  items.  The  dashed  line 
is  shifted  down  from  the  mean  by  four  standard  deviations.  Color  coded  symbols  in  Figure  3-14 
identify  anomalies  in  the  two-dimensional  mapped  data  above  and  below  this  threshold. 
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Figure  3-13  -  Scatter  plot  of  the  fit  mismatch  versus  coherence  ratio  for 
emplaced  UXO  in  the  GPO  area.  The  red  symbols  identify  anomalies  with  a  fit 
error  of  less  than  50%  and  fitted  depths  of  less  than  0.75  m.  The  blue  symbols 
identify  anomalies  that  exceed  these  thresholds.  See  text  for  discussion. 
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Figure  3-14  -  EMI  data  acquired  over  the  GPO  overlain  by  circles  identifying  seeded  ordnance 
items.  The  black  circles  identify  targets  with  fit  errors  less  than  50%  and  fitted  depths  less  than 
0.75  m.  The  yellow  circles  identify  seeded  UXO  that  exceed  these  thresholds. 
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Figure  3-15  shows  the  fit  error  versus  signal  strength  for  all  targets  in  the  GPO  and  PA1  subset 
survey  areas.  Here,  signal  strength  is  defined  as  QaVe,  the  average  quadrature  response  for  the 
middle  five  frequencies.  As  expected,  we  observe  larger  fit  errors  with  decreasing  signal 
strength.  Figure  3-16  superimposes  data  from  the  PA1  subset  area  on  Figure  3-13. 


Figure  3-15  -  Fitted  results  from  the  GPO  (diamond  symbols)  and  PA1  subset 
(x  symbols)  survey  areas.  Red  color  is  used  to  identify  high  SNR  targets  in  the 
GPO,  blue  is  low  SNR,  black  unknown,  and  cyan  identifies  targets  that  passed 
the  detection  threshold  but  were  later  rejected  based  on  statistical  rules. 

Fitted  results  in  the  feature  space  proposed  for  decision  making  are  shown  in  Figure  3-17.  Here, 
we  plot  the  fit  mismatch  versus  the  relative  coherence  ratio  normalized  by  the  standard  deviation 
about  the  mean  for  the  library  TOIs.  Anomalies  with  fit  errors  larger  than  50%  will  be  ranked 
Category  3  unless  rejected  by  our  statistical  rules  as  an  invalid  detection  pick.  For  anomalies 
with  fit  errors  less  than  50%,  we  propose  basing  the  discrimination  decision  on  the  coherence 
ratio  as  shown.  Because  the  library  consisted  of  TOI  only,  items  similar  to  those  in  the  library 
will  possess  ratios  close  to  one.  Objects  that  cannot  be  represented  by  the  TOI  polarizabilities 
have  ratios  less  than  one.  Based  in  part  on  trial  and  error,  we  selected  a  Category  1-2  threshold 
of  -4.  Based  on  these  limited  training  data,  this  threshold  captures  all  labeled  UXO  not  excluded 
due  to  low  SNR.  Figure  3-18  focuses  on  the  labeled  UXO  items  only  in  Figure  3-17  and  shows 
that  all  labeled  UXO  are  located  above  a  threshold  of  -2,  leaving  a  margin  of  safety  when  using  a 
threshold  of  -4. 
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Figure  3-16  -  Fitted  results  from  PA1  subset  area  are  superimposed  on  the 
fitted  results  from  the  GPO. 
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Figure  3-17  -  Scatter  plot  of  fit  results  for  anomalies  in  PA1  subset  and  GPO 
areas.  See  text  for  description. 

These  results  appear  promising,  but  cannot  be  used  to  predict  the  expected  performance  for  two 
reasons.  First,  the  training  data  do  not  include  any  non  UXO  objects.  Because  of  this  we  cannot 
say  anything  about  the  separation  between  TOI  and  non-TOIs.  Second,  we  are  not  confident  that 
the  distribution  of  labeled  data  in  the  GPO  adequately  reflects  the  distribution  of  unknown 
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targets  in  the  PA1  subset  area.  If  labeled  training  data  were  available  from  the  PA1  subset  area, 
the  discrimination  decision  criteria  discussed  above  could  be  further  refined.  Depending  on  the 
actual  distribution  of  UXO  in  the  PA1  subset  area,  it  might  be  possible  to  revise  the  GPO-based 
detection  threshold  upwards  and  thereby  reduce  the  number  of  items  placed  in  Category  3. 
Additionally,  the  detection  threshold  decision  could  also  be  revisited. 


Normalized  Coherence  Ratio 

Figure  3-18  -  Histogram  showing  the  frequency  of  labeled  UXO  as  a  function 
of  the  decision  metric.  As  shown,  all  labeled  UXO  not  excluded  based  in  SNR 
posses  values  greater  than  -2. 

3.2.10  Systems  Performance  and  Calibration  Item  Results 

For  the  GEMTADS  array,  the  standard  performance  checks  included  three  types  of 
measurements.  At  the  beginning  of  field  work  and  again  each  morning;  quiet,  static  data  were 
collected  for  a  period  (5  minutes  or  as  directed  by  the  QAO)  with  all  systems  powered  up  and 
wanned  up  (20  minutes  minimum).  Next,  two  calibration  items,  a  4”  diameter  Aluminum  (Al) 
sphere  and  a  fenite  rod  bundle,  were  placed  a  standard  distance  above  the  center  of  each  sensor 
coil  several  times  in  sequence  to  verify  the  response  of  each  sensor  to  each  object.  The  system 
was  kept  stationary  for  this  data  collection.  Finally,  a  systems  timing  check  using  a  fixed- 
position  chain  placed  on  the  ground  was  conducted.  The  timing  check  and  the  Al  sphere  and 
ferrite  measurements  were  repeated  at  the  end  of  each  survey  day. 

Two  pieces  of  data  are  extracted  from  the  stationary  data,  the  position  variation  and  the  GEM-3 
sensor  variation.  It  should  be  noted  that  the  sensor  platform  was  only  deployed  for  a  few  days, 
so  the  variations  reported  do  not  have  a  large  enough  sample  size  to  be  statistically  relevant. 
However,  the  reported  values  are  instructive,  taken  in  that  context.  The  2-D  position  variation 
was  evaluated  by  computing  the  standard  deviation  of  both  the  northing  and  easting  components 
of  the  position  data  for  the  entire  period  and  combining  them  as  the  square  root  of  the  sum  of  the 
squares.  The  3-D  position  variation  was  calculated  in  a  similar  manner  by  adding  the  vertical 
(HAE)  measurement.  The  standard  deviation  for  the  demedianed  sensor  data  from  each  sensor 
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was  computed  and  the  arithmetic  mean  was  computed  for  each  data  set.  The  aggregate  average 
and  standard  deviation  (la)  of  both  the  positioning  and  sensor  data  for  all  data  sets  were 
computed.  The  results  are  shown  in  the  following  time-series  figures.  Figure  3-19  shows  the 

2- D  position  variation  for  the  entire  demonstration,  which  is  summarized  in  tabulated  in  Table 

3- 9  along  with  the  3-D  variation. 
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Figure  3-19  -  2-D  position  variation  data  runs  for  stationary  data  collected 
near  the  base  camp.  The  horizontal  axis  is  survey  file  name.  The  solid  line 
represents  the  aggregate  average  positional  variation  and  the  dashed  lines 
represent  a  1  a  envelope. 


Table  3-9  -  FEW  Base  Camp  Stationary  Position  Variation  Results 


Result  Type 

Value 

2-D  Position 

0.61  ±0.11  cm 

3-D  Position 

0.93  ±  0.09  cm 

Figure  3-20  show  the  sensor  variation  from  the  stationary  data  collections  for  the  GEMTADS 
array.  Table  3-10  summarizes  the  stationary  GEM-3  sensor  data  collection  results.  The  quantity 
Qave  is  calculated  for  each  sensor  and  then  the  results  are  averaged  for  the  run. 


40 


Khadr  et  al. 


E 


CL 

CL 


Survey  File 


Figure  3-20  -  GEM-3  sensor  variation  for  stationary  data  collected  near  the 
base  camp.  The  horizontal  axis  is  survey  file  name.  The  solid  line  represents 
the  aggregate  average  positional  variation  and  the  dashed  lines  represent  ala 
envelope. 

Table  3-10  -  GEM-3  (GEMTADS)  Array  Static  Test 
Data  Results  (demedianed  values) 


Result  Type 

Value 

Qave 

0.193  ±0.014  ppm 

For  the  4”  dia.  Al  sphere,  the  sphere  was  placed  a  standard  distance  above  the  center  of  each 
sensor  several  times  to  measure  the  response  of  each  sensor.  Any  background  level  is  removed 
using  a  demedian  filter.  The  quantity  Qave  was  then  calculated  and  used  for  this  discussion.  The 
average  of  all  measurements  with  the  sphere  for  each  sensor  within  a  survey  is  calculated.  The 
average  amplitude  value  for  each  survey  /  sensor  pair  (average  and  standard  deviation  ( 1  a))  is 
tabulated  in  Table  3-11. 
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Table  3-1 1  -  Average  Demedianed  GEMTADS  Response  for  the  4-in  A1  Sphere 


Item 

Avg.  Signal 
(Qave,  ppm) 

Std.  Dev 

(Oave,  ppm,  la) 

4-in  A1  Sphere  Coil  1 

283.95 

5.3 

4-in  A1  Sphere  Coil  2 

285.18 

5.5 

4-in  A1  Sphere  Coil  3 

285.33 

4.4 

Figure  3-21  plots  the  average  response  amplitude  the  4-in  Aluminum  Sphere  for  all  data  sets  in  a 
time  series.  The  results  for  each  coil  are  shown  as  colored  symbols  using  the  standard  color 
coding.  The  variations  are  small  and  in  many  cases,  the  symbols  overlap.  The  solid  line 
indicates  the  aggregate  average  for  all  measurements  and  the  dashed  lines  indicate  ala 
envelope. 


Survey  File 


Figure  3-21  -  Overall  variation  of  GEMTADS,  Qave  value  for  daily  stationary 
data  collection.  The  horizontal  axis  is  survey  file  name.  The  solid  line  represents 
the  aggregate  average  sensor  variation  and  the  dashed  lines  represent  ala 
envelope.  The  symbols  represent  each  sensor  /  survey  data  pair.  The  symbol 
color  code  is  the  standard  one  described  previously  in  the  text. 

3.2.11  Demobilization 

At  the  end  of  field  operations,  all  NRL  equipment,  materials,  and  supplies  were  packed  in  the  53’ 
trailer.  A  government-contracted  transportation  company  picked  up  the  trailer  prior  to  the  field 
personnel  departing  the  site.  The  53’  trailer  was  delivered  to  Blossom  Point,  MD  as  indicated  in 
Table  3-4.  URS  had  previously  arranged  for  the  delivery  of  the  portable  toilet  and  arranged  for 
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the  removal  after  field  operations  were  completed.  The  shipping  container  was  left  in  place  for 
the  next  demonstrator  and  was  removed  when  no  longer  required. 

3.2.12  Health  and  Safety  Plan  (HASP) 

A  host  organization  exists  for  this  demonstration  site.  URS  Corporation,  Inc.  has  prepared  a 
Project  Health  and  Safety  Plan  [11]  and  an  Explosives  Safety  Submission  [9]  in  cooperation  with 
FEW  and  AFCEE  as  part  of  their  involvement  in  the  ongoing  RI.  This  demonstration  was 
conducted  under  the  authority  of  this  Health  and  Safety  Plan  in  coordination  with  URS  and 
FEW. 

3.3  Management  and  Staffing 

The  responsibilities  for  this  demonstration  are  outlined  in  Figure  3-22.  Dr.  Daniel  Steinhurst  is 
the  PI  of  this  project  and  filled  the  roles  of  Site  /  Project  Supervisor  and  Quality  Assurance 
Officer.  Mr.  Glenn  Harbaugh  was  the  Site  Safety  Officer  and  Data  Acquisition  Operator.  His 
duties  included  data  collection  and  safety  oversight  for  the  entire  team.  Dr.  Nagi  Khadr  is  the 
Data  Analyst  for  this  effort. 


Figure  3-22  -  Management  and  staffing  wiring  diagram 


GEMTADS  at  F.E.  Warren  AFB 


43 


4.  References 


1 .  “Moving  Platform  Orientation  for  an  Unexploded  Ordnance  Discrimination  System,”  D. 
Steinhurst,  N.  Khadr,  B.  Barrow,  and  H.  Nelson,  GPS  World,  2005, 16/5,  28  -  34. 

2.  “Airborne  MTADS  Demonstration  at  the  Badlands  Bombing  Range,  September,  2001,”  J.R. 
McDonald,  D.J.  Wright,  N.  Khadr,  H.H.  Nelson,  NRL/PU/61 10— 02-453. 

3.  “SAIC  Analysis  of  Survey  Data  Acquired  at  Camp  Sibert,  Interim  Letter  Report,  Version  2,” 
July,  2008.  http://serdp-estcp.org/content/download/5529/76987/fde/MM-0210 

4.  “Advanced  MTADS  Classification  for  Detection  and  Discrimination  of  UXO,”  H.H.  Nelson, 
T.H.  Bell,  J.R.  McDonald,  B.  Barrow,  NRL/MR-MM/61 10— 03-8663.  http://serdp- 
estcp.org/content/download/5568/77347/file/UX-4003-FR-01  .pdf 

5.  “Enhanced  UXO  Discrimination  Using  Frequency-Domain  Electromagnetic  Induction,” 
ESTCP  MM-0033  Final  Report,  May,  2007.  http://serdp- 

estcp .  org/content/ download/5  5  76/77411/  file/MM-003  3  -FR.pdf 

6.  “Survey  of  Munitions  Response  Technologies,”  ESTCP,  ITRC,  and  SERDP,  June,  2006. 
http://serdp-estcp.org/content/download/8565/104843/file/Survev  of  MR  Tech.pdf 

7.  “MTADS  Demonstration  at  Camp  Sibert  Magnetometer  /  EM61  Mkll  /  GEM-3  Arrays,” 

Demonstration  Data  Report,  G.R.  Harbaugh,  D.A.  Steinhurst,  N.  Khadr,  August,  2008. 
http  ://serdp-estcp  .org/content/ download/ 5614/77761/  file/MM-05  3  3  -GEM-Data.pdf 

8.  “ESTCP  Pilot  Program,  Classification  Approaches  in  Munitions  Response,”  H.H  Nelson,  K. 
Kaye,  A.  Andrews,  November  17,  2008.  http://serdp- 

estcp.org/content/download/7423/9482 1/version/l/file/ESTCP-Classification-CampSibert- 

FinalReport+ 12- 17-09.pdf 

9.  “Explosive  Safety  Submission  for  the  Closed  Base  Ranges,  F.E.  Warren  Air  Force  Base, 
Wyoming,”  prepared  by  URS  Group,  Inc.,  August  2004. 

10.  Range  Reconnaissance  Work  Plan,  F.E.  Warren  Air  Force  Base,  WY,  USAF,  September 
2003. 

11.  “Project  Health  and  Safety  Plan  for  Remedial  Investigation  at  the  Firing  Range  at  F.E. 
Warren  Air  Force  Base,  Cheyenne,  Wyoming,”  prepared  by  URS  Group,  Inc.,  January  2004. 


44 


Khadr  et  al. 


5.  Points  of  Contact 


ESTCP 

Anne  Andrews 

Katherine  Kaye 

F.E.  Warren  AFB 

John  Wright 

Brian  Powers 


NRL 

H.  H.  Nelson 


Nova  Research,  Inc. 

Dan  Steinhurst 


Glenn  Harbaugh 


Russell  Jeffries 


SAIC,  Inc. 

Nagi  Khadr 


Hotel  Accommodations 

La  Quinta  Inn  Cheyenne 


Program  Manager,  MM  Tel:  703-696-3826 

Fax:  703-696-2114 
anne.andrews@osd.mil 

Program  Assistant,  MM  Tel:  410-884-4447 

Fax:  703-478-0526 
kkaye@hgl.com 


Chief  of  Environmental  Restoration, 
F.E.  Warren  Air  Force  Base 

Group  Leader,  Environmental 

Investigation 

URS  Corporation 


Project  Manager,  WAA 


Principle  Investigator 


Site  Safety  Officer 


Logistics  Support 


Data  Analyst 


Tel:  307-773-4147 
j  ohn.wright@warren.af.mil 

Tel:  303-694-2770 
Fax:  303-694-3946 
Cell:  303-618-7395 
brian_powers@urscoip.com 


Tel:  202-767-3686 
Fax:  202-404-8119 
Cell:  202-215-4844 
herb.nelson@nrl.navy.mil 


Tel:  202-767-3556 
Fax:  202-404-8119 
Cell:  703-850-5217 
dan.steinhurst@nrl.navy.mil 

Tel:  301-392-1702 
Fax:  301-392-1702 
Cell:  804-761-5904 
roo749@yahoo.com 

Tel:  703-360-3900 
Fax:  703-360-3911 
Cell:  703-244-1245 
rj  effries@novaresearchonline.  com 


Tel:  703-413-0500 
Fax:  703-413-0505 
nagi  .khadr@saic  .com 


2410  W.  Lincolnway 
Cheyenne,  WY  82009-3301 
Tel:  307-632-7117 
Fax:  307-638-7807 


GEMTADS  at  F.E.  Warren  AFB 


45 


Appendix  A.  GEMTADS  Performance  at  the  Standardized  UXO 

Technology  Demonstration  Sites 

The  NRL  GEMTADS  array  was  developed  under  ESTCP  Project  MM-0033.  The  project 
objective  was  to  demonstrate  the  optimum  system  built  around  the  Geophex  GEM-3  EMI  sensor 
that  delivers  the  most  classification  performance  while  retaining  acceptable  survey  efficiency.  A 
three-sensor  array  system  was  designed  around  a  modified  GEM-3  sensor.  The  system  was  built 
and  characterized  and  then  demonstrated  at  the  Standardized  UXO  Demonstration  sites  at 
Aberdeen  Proving  Ground  and  Yuma  Proving  Ground.  At  each  of  the  sites,  the  Calibration 
Lanes,  the  Blind  Test  Grid,  and  as  much  of  the  Open  Field  Area  as  was  possible  were  surveyed. 
For  the  Blind  Test  Grid  and  the  Open  Field,  the  ranked  target  picks  were  submitted  to  Aberdeen 
Test  Center  for  scoring.  These  scoring  results  are  the  basis  for  characterizing  the  success  of  the 
demonstrations  and  the  performance  of  the  array.  Portions  of  Reference  5  are  reproduced  here  to 
summarize  the  performance  of  the  system. 

A.l  Aberdeen  Proving  Ground  Blind  Grid 

A.1.1  Response  Stage 

The  first  stage  of  scoring  at  the  Test  Sites  is  the  Response  Stage  where  anomalies  are  identified, 
or  detected.  For  this,  we  use  the  Qavg  quantity;  the  average  of  the  quadrature  response  for  the 
middle  five  frequencies. 


Qavg 


^  (0270ft:  +  0570ft:  +  0i23Oftz  +  0261Oftz  +  0543Oftz) 


We  choose  this  metric  because  of  the  lower  noise  in  the  quadrature  response  and  the  good  signal 
in  the  mid  frequencies  for  the  objects  of  interest.  A  Qavg  plot  for  the  APG  Blind  Grid  is  shown  in 
Figure  A-l.  The  400  cells  in  the  Blind  Grid  are  marked  with  white  squares  in  Figure  A-l.  A 
summary  of  the  GEM  array  detection  performance  is  given  in  Table  A-l. 

Table  A-l  -  Summary  of  Detection  Performance  at  the  APG  Blind  Grid. 


Cell  Contents 

Number  of 
Cells 

Number 

Correct 

Number 

Incorrect 

Single  Ordnance  Item 

84 

73 

11 

Ordnance  Item  with  Clutter 

7 

7 

0 

Single  Clutter  Item 

95 

91 

4 

Two  Clutter  Items 

8 

8 

0 

“Empty” 

206 

174 

32 

Total 

400 
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Figure  A-l  -  Qavg  anomaly  image  map  of  the  APG  Blind  Grid 

The  32  cells  reported  as  “Empty”  but  for  which  we  made  a  declaration  require  some  discussion. 
Only  12  of  these  false  positives  showed  signal  in  the  GEM  array  survey  only.  Seven  of  these 
cells  had  a  detection  by  the  GEM  array,  the  EM61  HH,  and  the  magnetometer  array.  Ten  had  a 
detection  by  the  GEM  array  and  the  EM6 1  HH  and  3  had  a  detection  by  the  GEM  array  and  the 
magnetometer  array.  An  example  of  this  is  cell  B8  which  is  highlighted  in  Figure  A-l.  It  is 
difficult  to  understand  the  observed  signal  unless  there  is  some  inadvertent  metal  in  this  cell. 

An  indication  of  the  depth  performance  of  the  system  is  shown  in  Figure  A-2.  The  detected 
items  are  shown  as  black  triangles  and  the  missed  items  are  shown  as  red  crosses.  The  reference 
line  corresponds  to  a  depth  of  1  lx  the  item  diameter.  As  can  be  seen,  the  GEM  array  is  capable 
of  detecting  targets  down  to  and  below  1 1  times  their  diameter  at  this  site. 
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Figure  A-2  -  Qavg  Detection  performance  as  a  function  of  depth  at  the 
APG  Blind  Grid 

The  response  stage  data  are  plotted  in  Figure  A-3  and  Figure  A-4  in  the  manner  of  the 
Standardized  Test  Site  scoring  reports.  Figure  A-3  shows  cumulative  ordnance  count  vs. 
cumulative  clutter  count.  Since  the  targets  are  ordered  by  signal  amplitude  at  the  response  stage 
it  is  no  surprise  that  this  plot  is  essentially  along  the  diagonal. 


Figure  A-3  -  Response  stage  results  showing 
cumulative  ordnance  count  vs.  cumulative  clutter 

A  better  measure  of  system  capability  is  shown  in  Figure  A-4  which  plots  cumulative  occupied 
cells  vs.  adjusted  cumulative  blank  cells.  Cells  such  as  B8  which  obviously  contain  buried  metal 
were  excluded  from  the  blank  count. 
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Figure  A-4  -  Response  stage  performance 
showing  cumulative  occupied  cell  count  plotted 
vs.  adjusted  cumulative  blank  cell  count 

A.1.2  Discrimination  Stage 

The  prescription  for  discrimination  using  GEMTADS  is  to  compare  the  measured  response  of  a 
target  to  each  of  a  set  of  library  response  functions  in  turn,  and  to  determine  which  library  item 
results  in  the  closest  match  to  the  target.  If  the  match  is  good  enough,  we  can  declare  the  target 
as  being  that  member  of  the  library.  To  quantify  the  “goodness-of-fif ’,  we  compute  the  x  for 
the  each  match.  For  the  APG  Blind  Grid,  three  different  methods  of  discrimination  based  on  the 
X  were  evaluated. 

The  first  method  is  one  based  on  the  weighing  of  the  x  by  the  signal  amplitude.  If  the  signal 
level  for  the  various  targets  differs  by  a  large  amount  (some  targets  quite  shallow  and  some  very 
deep)  the  computed  x~  can  be  strongly  affected  by  the  signal  amplitude.  To  test  this  possibility, 
we  computed  the  x  for  the  best  match  weighting  the  data  by  the  usual  l/rms“  (where  the  rms 
deviation  is  detennined  from  areas  between  targets)  and  by  l/(rms  +  0.01  x  signal)  in  an  attempt 
to  reduce  the  influence  of  depth  on  the  computed  The  calculated  with  the  signal-based 
weighting  was  used  for  our  declarations  at  the  APG  Blind  Grid.  Based  on  the  results  from  the 
Calibration  Lanes  (which  was  all  we  had  available  at  the  time),  we  established  a  x2  threshold  of 
0.01  for  the  ordnance/clutter  decisions.  This  is  a  little  less  than  three  standard  deviations  above 
the  ordnance  mean.  The  reported  Discrimination  Response  Factor  was  just  the  inverse  of  the  x“- 

The  second  method  assumes  that  this  strong  variation  of  %2  with  signal  amplitude  arises  from  the 
bouncing  motion  of  the  sensor  array  as  it  traverses  the  rough  field.  Over  a  high-signal  target, 
small  variations  in  z  result  in  relatively  large  variations  in  signal  as  compared  to  over  a  deep, 
low-signal  target.  In  this  case,  we  can  model  the  bouncing  noise  by  (K  *  signal)  and  the  correct 
weighting  would  be  l/(nns2  +  (K  *  signal)2).  Based  on  data  collected  on  the  Blossom  Point  Test 
Field,  at  values  of  K  around  0.3,  the  scaling  of  x  with  signal  amplitude  seems  to  flatten  out. 
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Scaling  the  weights  by  the  signal  improves  the  perfonnance  of  the  discrimination  but  is  not  very 
practical  as  the  scaling  coefficient  is  determined  after  the  fact.  For  the  later  YPG  and  APG  Open 
Field  demonstrations  we  employed  another  method  to  mitigate  the  effects  of  bouncing  noise. 
Each  target  was  fit  using  a  full,  unconstrained  3-[]  model  as  well  as  the  library  model.  The  ratio 
of  the  x2  for  these  two  methods,  which  eliminates  the  dependence  on  signal  amplitude,  should 
approach  1  if  the  item  is  in  the  library. 

The  ROC  curves  for  the  application  of  these  three  discrimination  methods  to  the  APG  Blind  Grid 
are  shown  in  Figure  A-5  through  Figure  A-7.  The  standard  x  weighting  (Figure  A-5)  and  the 
modified  weighting  with  “bounce  noise”  added  (Figure  A-6)  result  in  curves  that  vary  little  from 
the  chance  diagonal.  There  are  fewer  items  in  Figure  A-6  than  in  Figure  A-5.  The  original 
submission  to  APG  required  that  a  discrimination  score  be  included  for  all  cells,  even  those 
below  our  detection  threshold.  We  arbitrarily  assigned  these  cells  a  low  discrimination  score. 
The  x  with  “bouncing  noise”  analysis  was  only  applied  to  cells  in  which  we  declared  a 
detection. 


Figure  A-5  -  ROC  curve  for  the  yj  weighting 
applied  to  the  APG  Blind  Grid  as  shown  in  the  left- 
hand  side  of  Figures  25  and  26  of  Reference  5 
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Figure  A-6  -  ROC  curve  for  the  case  of  %2 
weighting  with  an  estimate  of  "bouncing  noise" 
included  applied  to  the  APB  Blind  Grid 

The  x  ratio  method  (Figure  A-7)  does  show  some  promise.  Notice,  however,  that  the  curve  in 
Figure  A-7  includes  even  fewer  ordnance  and  clutter  items  than  in  Figure  A-6.  The  x  ratio 
method  requires  two  different  inversions  to  converge  to  sensible  results  in  order  to  calculate  the 
ratio.  As  the  signal-to-noise  ratio  decreases,  this  becomes  an  increasingly  difficult  hurdle. 
Library  methods  such  as  this  can  work  well  when  the  expected  targets  are  well  defined  but  can 
provide  inappropriate  results  when  a  munitions  item  not  in  the  library  is  encountered. 


Figure  A-7  -  ROC  curve  for  the  %  ratio  method 
applied  to  the  APG  Blind  Grid 
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A.2  Aberdeen  Proving  Ground  Open  Field 

Selected  results  from  our  surveys  at  the  Open  Field  at  the  Aberdeen  Proving  Ground 
Standardized  Test  Site  have  been  provided  to  us  by  analysts  at  the  Institute  for  Defense 
Analyses.  These  results  are  summarized  graphically  in  the  following  sections. 

A.2.1  Response  Stage 

Response  stage  data  from  the  Open  Field  scenario  at  the  APG  Standardized  Test  Site  is  shown  in 
Figure  A-8  as  a  plot  of  probability  of  detection  vs.  normalized  background  alarm  rate.  There  are 
two  analysis  models  shown  on  the  plot.  The  first,  the  red  line,  corresponds  to  considering  only 
those  targets  that  were  covered  by  the  survey  and  are  not  within  2  m  of  another  target.  The 
analysis  corresponding  to  the  blue  line  retains  those  limitations  and  also  excludes  those  targets 
deeper  than  1  lx  their  diameter.  We  showed  in  Figure  20  of  Reference  5  that  the  GEM  array 
demonstrated  is  able  to  detect  small  and  medium  targets  below  this  relative  depth  but  our 
detection  efficiency  falls  off  at  depths  below  1  lx  the  item  diameter.  Response  stage  results 
broken  out  by  item  type  are  shown  in  Figure  A-9.  In  this  figure,  the  depth  of  100%  detection  is 
denoted  by  the  blue  bar  and  the  depth  of  maximum  detection  is  shown  as  the  horizontal  line.  For 
a  number  of  the  items,  105-mm  HEAT  for  example,  these  two  depths  are  the  same.  For  the 
majority  of  the  items,  the  maximum  depth  of  detection  is  below  the  depth  of  100%  detection. 
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Figure  A-8  -  Detection  performance  at  the  APG  Open  Field 
Scenario.  The  red  line  is  derived  considering  only  targets  that  were 
covered  in  the  survey  and  are  not  within  2  m  of  another  target.  The 
blue  line  retains  those  criteria  and  also  excludes  targets  deeper  than 
1  lx  their  diameter. 
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Figure  A-9  -  Response  stage  results  for  the  APG  Open  Field 
scenario  broken  out  by  target  type 

A.2.2  Discrimination  Stage 

Discrimination  stage  performance  at  the  APG  Open  Field  using  the  same  two  analysis  models  is 
shown  in  Figure  A-10.  As  above,  the  exclusion  of  items  at  depths  below  1  lx  their  diameter 
(presumably  lower  S/N  anomalies)  improves  the  discrimination  performance  obtained. 


Figure  A-10  -  Discrimination  performance  at  the  APG  Open 
Field  Scenario.  The  red  line  is  derived  considering  only 
targets  that  were  covered  in  the  survey  and  are  not  within  2  m 
of  another  target.  The  blue  line  retains  those  criteria  and  also 
excludes  targets  deeper  than  1  lx  their  diameter. 
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A.3  Yuma  Proving  Ground  Open  Field 

Selected  results  from  our  surveys  at  the  Open  Field  at  the  Yuma  Proving  Ground  Standardized 
Test  Site  have  been  provided  to  us  by  analysts  at  the  Institute  for  Defense  Analyses.  These 
results  are  summarized  graphically  in  the  following  sections. 

A.3.1  Response  Stage 

Response  stage  results  for  the  YPG  Open  Field  scenario  are  shown  in  Figure  A-l  1  and  Figure 
A- 12.  As  for  APG,  they  are  analyzed  by  excluding  first  items  that  were  not  covered  by  the 
survey  or  are  within  2-m  of  another  item  then  retaining  those  exclusions  and  further  excluding 
items  deeper  than  1  lx  their  diameter.  Notice  that  the  background  alarm  rates  in  Figure  A-l  1  are 
more  than  a  factor  of  two  smaller  than  the  corresponding  results  from  Aberdeen.  Although  the 
Yuma  site  is  more  geologically  active  than  Aberdeen,  it  is  smoother  so  there  were  fewer  false 
alarms  due  to  platform  bouncing  over  deep  ruts.  Detection  depths  at  Yuma  are,  in  general,  in 
line  with  those  obtained  at  Aberdeen.  Note  however,  that  a  shallow  bomb  was  apparently  missed 
resulting  in  an  unusual  plot  for  that  target  type. 
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Figure  A-l  1  -  Detection  performance  at  the  YPG  Open  Field 
Scenario.  The  red  line  is  derived  considering  only  targets  that  were 
covered  in  the  survey  and  are  not  within  2  m  of  another  target.  The 
blue  line  retains  those  criteria  and  also  excludes  targets  deeper  than 
1  lx  their  diameter. 

A.3.2  Discrimination  Stage 

Discrimination  Stage  results  from  the  YPG  Open  Field  are  shown  in  Figure  A-13.  As  before, 
exclusion  of  items  that  are  deeper  than  1  lx  their  diameter  improves  performance  which  is  better, 
on  the  whole,  than  that  observed  at  Aberdeen.  As  with  the  response  stage,  this  is  likely  due  to 
the  lower  platform  motion  noise  observed  at  the  Yuma  site. 
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Figure  A- 12  -  Response  stage  results  for  the  YPG  Open  Field 
scenario  broken  out  by  target  type 


Figure  A- 13  -  Discrimination  performance  at  the  YPG  Open  Field 
Scenario.  The  red  line  is  derived  considering  only  targets  that 
were  covered  in  the  survey  and  are  not  within  2  m  of  another 
target.  The  blue  line  retains  those  criteria  and  also  excludes  targets 
deeper  than  1  lx  their  diameter. 
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Appendix  B.  Quality  Assurance  Project  Plan  (QAPP) 

B.l  Purpose  and  Scope  of  the  Plan 

The  collection  and  archiving  of  high  quality  survey  data  in  auditable  and  defensible  manner  is 
critical  to  insure  the  credibility  of  the  data  collected  and  to  support  decisions  based  in  part  or  in 
total  on  that  data.  This  Appendix  outlines  the  standard  process  used  in  the  NRL  MTADS 
program  to  collect  survey  data,  conduct  quality  checks  to  insure  the  quality  of  the  data,  and  then 
process  and  archive  the  data.  With  the  exception  of  Section  B.9,  the  discussion  focuses  on  the 
magnetometer  array  system.  For  the  EM61  Mkll  and  GEMTADS  sensor  systems  similar 
procedures  are  used,  different  only  in  the  details  of  the  data  collected  for  each  sensor  system. 

Any  sensor  platform  unique  items  are  indicated  where  appropriate. 

B.2  Quality  Assurance  Responsibilities 

The  team  as  a  whole  is  involved  in  insuring  the  quality  of  collected  data.  The  MTADS  has  been 
designed  to  provide  a  series  of  visual  indicators  to  the  operator  regarding  the  status  of  the 
individual  subsystems  that  comprise  the  MTADS.  The  operator  is  responsible  for  monitoring 
these  indicators  and  halting  data  collection  immediately  if  any  problems  are  indicated.  The  issue 
will  be  resolved  prior  to  resuming  operations.  All  team  members  are  involved  in  visual  walk- 
around  inspections  of  the  system  at  least  daily.  For  each  survey  file  set,  the  data  preprocessing 
tasks  are  logging  receipt  of  the  file  set,  archiving  the  file  set,  verifying  that  all  files  within  the  file 
set  are  valid,  and  verifying  that  each  sensor  channel  contains  valid  data  with  sufficient  SNR 
(where  appropriate).  Any  section  of  data  which  is  found  lacking  is  flagged  accordingly  and  not 
processed  any  further.  The  section  will  be  logged  for  future  re-acquisition  if  necessary.  The  data 
analyst  is  responsible  for  the  data  preprocessing  and  processing  tasks  with  the  site  /  project 
manager’s  assistance  as  available.  Dr.  Daniel  Steinhurst  will  serve  as  the  Quality  Assurance 
Officer  for  this  project. 

B.3  Data  Quality  Parameters 

Incoming  survey  data  will  be  evaluated  for:  completeness  of  the  data  set,  location  quality  for  the 
data  set,  and  for  proper  operation  of  the  magnetometer  sensors.  The  following  section  details  in 
an  example  how  the  data  quality  issues  are  addressed  throughout  the  survey. 

B.4  Calibration  Procedures,  Quality  Control  Checks,  and  Corrective  Action 

The  following  procedure  constitutes  a  typical  startup  for  the  MTADS  system  for  both  initial 
startup  and  as  daily  system  evaluations.  The  RTK  GPS  base  station  receiver  and  radio  link  will 
be  established  on  one  of  the  established  control  points.  The  validity  of  the  control  point  location 
will  be  verified  using  the  MTADS  man-portable  RTK  GPS  rover  receiver  to  occupy  one  or  more 
of  the  established  control  points  using  the  control  point  occupied  by  the  GPS  base  station  as  a 
reference  as  required  by  the  QAO. 

For  the  GEMTADS  array,  the  standard  performance  checks  include  three  types  of  measurements. 
Initial  system  startup,  at  the  beginning  of  field  work  and  again  each  morning,  consists  of  three 
measurements.  First,  quiet,  static  data  are  collected  for  a  period  (15-20  minutes  or  as  directed 
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by  the  QAO)  with  all  systems  powered  up  and  warmed  up  (typically  30  minutes).  Next,  two 
calibration  items,  a  4”  diameter  Aluminum  (Al)  sphere  and  a  ferrite  rod  bundle,  are  placed  a 
standard  distance  above  the  center  of  each  sensor  coil  several  times  in  sequence  to  verify  the 
response  of  each  sensor  to  each  object.  The  system  is  stationary  for  this  data  collection.  Finally, 
a  systems  timing  check  using  a  fixed-position  wire  or  chain  placed  on  the  ground  is  conducted. 

At  the  discretion  of  the  QAO,  the  timing  check  may  be  repeated  in  the  middle  of  the  survey  day. 
At  the  discretion  of  the  QAO,  the  timing  check  and  the  Al  sphere  and  ferrite  measurements  may 
be  repeated  at  the  end  of  the  day.  When  all  system  checks  are  completed  to  the  satisfaction  of 
the  QAO,  the  survey  will  commence. 

For  the  EM61  Mkll  array,  the  standard  perfonnance  checks  are  the  same  as  for  the  GEMTADS 
with  the  ferrite  rod  measurements  deleted.  The  ferrite  rod  is  not  a  useful  calibration  item  for  this 
time-domain  instrument.  For  the  magnetometer  array,  the  Al  sphere  measurements  are  also 
deleted  and  the  quiet  period  is  reduced  to  5-10  minutes.  Each  sensor  platform’s  performance 
check  requirements  are  based  on  data  rates  and  the  historical  stability  and  reproducibility  of  each 
sensor  type. 

Preventative  maintenance  inspections  will  be  conducted  at  least  once  a  day  by  all  team  members, 
focusing  particularly  on  the  tow  vehicle  and  sensor  trailer.  Any  deficiencies  will  be  addressed 
according  to  the  severity  of  the  deficiency.  Parts,  tools,  and  materials  for  many  maintenance 
scenarios  are  available  in  the  system  spares  inventory  which  will  be  on  site.  Status  on  any  break¬ 
downs  /  failures  which  will  result  in  long-term  delays  in  operations  will  be  immediately  reported 
to  a  representative  of  the  Program  Office. 

MTADS  survey  raw  data  generally  falls  into  two  categories,  location  and  sensor  measurements. 
For  the  magnetometer  array,  the  data  set  is  comprised  of  ten  separate  files,  each  containing  the 
data  from  a  single  system  device.  Each  device  has  a  unique  data  rate.  A  software  package 
written  by  NRL  examines  each  file  and  compares  the  number  of  entries  to  the  product  (total 
survey  time  *  data  rate).  Any  discrepancies  are  flagged  for  the  data  analyst  to  address.  For 
magnetometer  sensor  data,  operational  values  are  typically  on  the  order  of  50,000  nT  and  have 
noise  levels  of  ~0.5  nT  peak-to-peak  (PP)  static  and  3-5  nT  PP  in  motion.  Sensor  “drop-outs” 
can  occur  if  the  sensor  is  tilted  out  of  the  operation  zone  with  respect  to  the  earth’s  magnetic 
field.  If  a  sensor  cable  is  severed  or  damaged  while  in  motion,  the  sensor  output  value  will  drop 
below  20,000  nT  and/or  become  very  noisy  (1,000’s  of  nT  PP).  All  magnetometer  sensor 
channels  (8  total)  are  examined  in  each  survey  file  set  for  these  conditions  and  any  data  which  is 
deemed  unsatisfactory  is  flagged  and  not  processed  further.  For  location  data,  the  RTK  GPS 
receivers  present  a  Fix  Quality  value  that  relates  to  the  quality  /  precision  of  the  reported 
position.  A  Fix  Quality  (FQ)  value  of  3  (RTK  Fixed)  is  the  best  accuracy  (typically  3-5  cm  or 
better).  A  FQ  value  of  2  (RTK  Float)  indicates  that  the  highest  level  of  RTK  has  not  be  reached 
yet  and  location  accuracy  can  be  degraded  to  as  poor  as  -1  m.  FQ  1  &  4  correspond  to 
Autonomous  and  DGPS  operational  modes  respectively.  Data  collected  under  FQ  3  and  FQ  2  (at 
the  discretion  of  the  data  analyst)  are  retained.  Any  other  data  are  deemed  unsatisfactory, 
flagged  and  not  processed  further.  The  survey  section  containing  the  flagged  data  will  be  logged 
for  future  re-acquisition  if  required.  Data  which  meet  these  standards  are  of  the  quality  typical  of 
the  MTADS  system. 
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For  the  EM61  Mkll  array,  stationary  sensor  noise  levels  were  approximately  1  mV  for  all  time 
gates  at  the  Fonner  Camp  Sibert  ESTCP  UXO  Discrimination  Study  demonstration  [7]  and  the 
dynamic  background  signal  levels  on  the  calibration  lane  were  1.6  ±  0.1  mV,  si.  The 
corresponding  GEMTADS  results  were  0.232  ±0.177  ppm,  Qave  for  stationary  measurements 
and  the  dynamic  background  level  was  0.35  ±  0.06,  Qave  for  the  calibration  lane.  Also  note  that 
for  the  EM61  Mkll  and  GEMTADS  arrays,  the  orientation  of  the  platform  is  also  recorded  as 
described  in  Section  2.  Two  additional  GPS  strings  of  the  “PTNL,  AVR”  type  are  recorded  and 
the  output  of  the  IMU  is  also  recorded.  Similar  quality  checks  are  applied  to  these  additional 
data  sources. 

B.5  Demonstration  Procedures 

See  Section  B.4.  The  same  discussion  applies  to  this  section. 

B.6  Calculation  of  Data  Quality  Indicators 

There  are  no  specialized  equations  required.  The  methods  are  outlined  in  Section  B.4. 

B.7  Performance  and  System  Audits 

See  Section  B.4.  The  same  discussion  applies  to  this  section. 

B.8  Quality  Assurance  Reports 

The  results  of  the  daily  system  checkout  runs  for  the  standard  systems  checks  and  the  dynamic 
survey  of  the  emplaced  items  will  be  reported  to  the  QAO  daily.  The  Data  Analyst  will  report 
any  data  sections  requiring  reacquisition  to  the  site  /  project  manager  for  a  given  day  by  the  start 
of  work  the  following  morning. 

B.9  GEM-3  (GEMTADS)  Array  Data  File  Formats 

Each  survey  file  set  contains  eight  files  which  constitute  the  ‘raw  data’.  The  file  name  structure 
is  based  on  the  date  and  time  of  the  start  logging  event  (MMMDDYYYY_HHMMSS).  Data  in 
each  file  is  time  stamped  with  the  appropriate  clock  to  allow  synchronization  between  files. 


MMMDD YYYY_HHMMS  S . Survey . AVR1 . csv 

GPS  output,  Trimble  PTNL,  GGK  sentence  at  20  Hz  (position),  PTNL, AVR  sentence 
for  MB1  /  MB2  receiver  pair  at  10  Hz  (orientation) . 

MMMDD YYYY_HHMMSS . Survey . AVR2 . csv 

GPS  output,  Trimble  PTNL, AVR  sentence  for  MB2  /  MR  receiver  pair  at  10  Hz 
(orientation) . 

MMMDD YYYY_HHMMS  S_ID 1 . Survey . GEM . csv 

Output  from  Sensor  #1  (Port,  Forward)  ,  In-phase  and  Quadrature  data  for  9 
frequencies . 

MMMDDYYYY_HHMMS  S_ID2 . Survey . GEM . csv 

Output  from  Sensor  #2  (Center,  Aft) ,  In-phase  and  Quadrature  data  for  9 
frequencies . 

MMMDD YYYY_HHMMS  S_ID  3 . Survey . GEM . csv 

Output  from  Sensor  #3  (Starboard,  Forward),  In-phase  and  Quadrature  data  for  9 
frequencies . 

MMMDD YYYY_HHMMSS . Survey . PPS . csv 

pulse  per  second  (PPS)  from  GPS  receiver,  1  Hz. 

MMMDD YYYY_HHMMSS . Survey . IMU . tbf 
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Output  from  IMU  (pitch,  roll,  angular  rates,  accelerations,  etc.)  in  packed 
binary  format. 

MMMDD YYYY_HHMMS  S . Survey . UTC . csv 

UTC  time  tag  from  GPS  receiver  MR,  "The  time  will  be"  message  for  next  PPS,  1 
Hz. 

The  IMU  data  are  recorded  in  a  packed  binary  data  format  with  a  time  tag  appended  periodically. 
The  data  packet  format  is  described  in  the  manufacturer’s  manuals  and  technical  notes  and  is  not 
reproduced  here.  The  GEM-3  data  files  are  well  annotated  internally  and  the  fonnat  is  not 
discussed  here. 

Located  data  archives  are  stored  in  an  ASCII  file  format  of  the  form: 

For  located,  demedianed  GEM-3  data: 

T  UTC  time  in  seconds  past  midnight 

X  (UTM  Zone  X,  NAD83,  m)  Easting  for  GPS  antenna 

Y  (UTM  Zone  X,  NAD83,  m)  Northing  for  GPS  antenna 

Z  (WGS84,  m)  Height  above  Ellipsoid  for  GPS  antenna 

COG  (degrees)  Course  over  ground  (Heading)  of  array  in  Grid  North 

frame.  North  =  0  degrees 

GPS  Roll  (degrees)  Roll  of  the  sensor  platform  from  GPS  array 
GPS  Pitch  (degrees)  Pitch  of  the  sensor  platform  from  GPS  array 
IMU  Roll  (degrees)  Roll  of  the  sensor  platform  from  IMU 
IMU  Pitch  (degrees)  Pitch  of  the  sensor  platform  from  IMU 
SensorlD  Denotes  which  sensor  data  are  from 

I  90Hz  (ppm)  Demedianed  GEM-3  data  for  90  Hz,  In-phase  Response 

Q  90Hz  (ppm)  Demedianed  GEM-3  data  for  90  Hz,  Quadrature  Response 


I  20010Hz  (ppm)  Demedianed  GEM-3  data  for  20,010  Hz,  In-phase  Response 

Q  20010Hz  (ppm)  Demedianed  GEM-3  data  for  20,010  Hz,  Quadrature  Response 

Filename  Filename  of  source  data  file 

where  X  is  the  appropriate  UTM  zone  (13N  for  Cheyenne,  WY) 

B.10  Data  Storage  and  Archiving  Procedures 

Data  are  stored  electronically  as  collected  on  the  MTADS  vehicle  DAS  computer  hard  drive. 
Approximately  every  two  survey  hours,  the  collected  data  are  copied  onto  magnetic  disks 
(Iomega  ZIP  250)  or  removable  media  and  transferred  to  the  data  analyst.  The  data  are  moved 
onto  the  data  analyst’s  computer  and  the  media  is  recycled.  Raw  data  and  analysis  results  are 
backed  up  from  the  data  analyst’s  computer  to  optical  media  (CD-R  or  DVD-R)  or  external  hard 
disks  daily.  These  results  are  archived  on  an  internal  file  server  at  NRL  at  the  end  of  the  survey. 
All  field  notes  /  activity  logs  are  written  in  ink  and  stored  in  archival  laboratory  notebooks. 
These  notebooks  are  archived  at  NRL  or  SAIC.  Relevant  sections  are  reproduced  in 
demonstration  reports.  Dr.  Daniel  Steinhurst  is  the  POC  for  obtaining  data  and  other 
information.  His  contact  infonnation  is  provided  in  Section  5  of  this  report. 


